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ABSIEACT 

This booklet deals vith the practical steps of 
educational evaluation: who should negotiate the con+ract? Who 
initiates evaluation? What are "goals process" and "parts process" 
and how are they matched? What are the steps in putting the prqcess 
of evaluation into operation? What are the criteria for assessing 
observational techniques? What data need to be collected? Once a 
decision-'maker has a report of evaluation, what will he do with it? 
suppose a school district has limited resources, what can the 
decision-maker do? A glossary is provided at the end of the booklet 
so that the reader will kno?^ what the writer means by some words or 
terms used. This guide will be followed by booklets addressed to 
specific audiences: e.g., members of the boards of education, 
administrators, teachers and parents. (BBJ 
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Preface 

This booklet has been prepared as a guide for decision- 
makers in education (Board ^nembers , adrain istrators and teachers* . .) 
who may hire an evaluator to begin an evaluation. The user will 
find it a very helpful manual in delineating evaluation problems* 

After dealing with some basic concepts of evaluation to 
clarify misunderstandings and misinformation, Dr. Benedict, who 
prepared this booklet, deals with the practical steps of evaluation: 
Who should negotiate the contract? Who initiates evaluation? 
What are "goals process'* and "parts procesc;'* and how are they 
matched? What are the steps in putting the process of evaluation 
into operation? What are the criteria for assessing observational 
techniques? What data need to be collected? Once a decision- 
maker has a report of evaluation, what will he do with it? Suppose 
a school district has limited resource?^, what can the decision- 
maker do? 

Dr. Benedict has tried, and I believe he has succeeded, to 
avoid some of the educational terminology that has *'fu22y** meaning. 
However, a glossary is provided at the end of the booklet so that 
the reader will know what the writer means by some words or terms 
used . 

This guide will be followed by booklets addressed to specific 
audiences: e.g., members of the boards of education, administrators, 
teachers and parents. 

In introducing this guide. Project Evaluation, Capitol Region 
Education Council, considers it a step on the long road of evalu- 
ation. I hope that it will be v;idely used among decision-makers 
towards the betterment of the educational process. 

Philip S. Saif, Director 
Pro j ect Evaluation 
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I. AN INTRODUCTION TO EDUCATIONAL EVALUATION 

The starting point in evaluation occurs well before the 
evaluation begins. That point should be when one asks, and 
answers, the question: ''Why do I want to evaluate?'^ Unless this 
question is answered, an evaluation should not be undertaken be- 
cause, in fact, maybe it is not evaluation that is needed or 
wanted, but something else. 

Here are some typical reasons for wanting to have an evaluation: 

(1) For public relations - so someone will like me, or fund 
me , etc. 

(2) To find out what the students need. 

(3) To make program or planning decisions. 

(4) To provide systematic, ongoing information (data) as a 
basis for making decisions- 

However, not all of these are evaluation, so a decision-maker 
would not (should not) hire an evaluator to do all of these- For 
example, evaluation is fundamentally different from a public 
relations job. PR brings to mind Madison Avenue, marketing, public 
image and so on. This is not to say that a PR man might not want 
to avail himself of some of the data an evaluation design would 
collect. This is to say, however, that the evaluation designer's 
job is not PR. If an enterprise wishes to sell itself to the 
public, it hires a PR expert, goes to an advertising agency or 
buys commercial time. If an enterprise desires objective, system- 
atic feedback about the status of that enterprise, it hires an 
evaluator or evaluation designer. 

It is important not to confuse the roles of PR and evaluation.^ 
for the methods, nature and goals of each are fundamentally dif- 
ferent. A PR expert is in a much better position to do a much 
better job of promoting one's image or selling one's wares than is 
a person trained only in evaluation. Conversely, a PR man is not 
usually equipped or skilled in evaluation design. Basically, then, 
this simple rule of thumb should be remembered: if one wants a 
PR job, hire a PR man; if one wants an evaluation design, hire 
an evaluator. 
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The same can be said of Purpose §2. This purpose really 
demands a needs analysis expert, not a person skilled in evaluation. 
While the two may be similar, a needs analysis can be better done 
by someone trained in such procedures, rather than someone trained 
in evaluation. 

Purpose ^^3 above is also not evaluation, Makinr program or 
planning decisions is decision making. If an enterprise wants to 
hire someone to make decisions for them, to improve their decision 
making, to insure that the enterprise makes *'good" or ''the right^' 
decisions, then the enterprise should hire someone trained in 
decision making. 

The fourth purpose is the one being agreed upon by more and 
more 'evaluation experts". Evaluation has as its primary purpose 
the collection of data to be used as feedback to decision-makers 
in order to provide a basis for decision making, not to make 
decisions. It is more than assessing student achievement, more 
than measuring the percentage of achievement of an instructional 
objective. Rather, evaluation should be the collection of speci- 
fic data about a given program or project which the decision-makers 
of that project want or that the enterprise deems important and 
which will be used by those decision-makers for decision making 
regarding the strengths and weaknesses of their particular 
enterprise . 
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Review : An Introduction to Educational Evaluation 

(1) The first step before an evaluation is bep:un is to 
determine the purpose for conducting it, 

(2) If your purpose is to have data for decision makinp:, 
then you are in the same ball park as educational 
evaluation experts (Cronbach, Guba, Stuff lebeam. 
Fortune, Hutchinson, Vorthen, Provus, and tnany others). 

(3) If your purpose is not to collect data for your de- 
cision makinip^ needs, but some other purpose, seek an 
expert in that ball park. 

Having come up with an answer to "Why do I want to evaluate? 
the next step is to consider some basic concepts of evaluation. 

/ 



Some Basic Concepts of Evaluation 

The term "evaluation'* is an all-encompassing concept in 

education today* Many, many processes are termed 'evaluation'' 

when in fact they would probably be better termed something else. 

Some examples will show how fuzzy a concept "evaluation'' is. 

The testing of products to describe their characteristics 
is called evaluation. Why not simply call it product testing ? 
The accumulation of data about an institution's operation - 
its income, expenditures, costs per credit hour, faculty- 
student ratio, etc. is called evaluation. V/hy not simply 
call it institutional accounting ? 

The measurement of pupils' knowledge at the beginning and 
end of a course is called evaluation, Why not simply call 
it achievement testing ? (Pace, 1968, pp. 1-2) 
These are a few examples which show some of the different things 
called evaluation. Yet each of these is not evaluation. Evalu- 
ation is different. The purpose of this section is to discuss 
what is and is not evaluation. 

Traditionally, evaluation has been conceived of as the 
administering of a test, usually standardized, for the purpose 
of determining something, usually student achievement. Or 
secondly, evaluation has been traditionally conceived of as 
determining "how good" or ''how bad^' something stacks up to some- 
thing else, i.e.. Program A to Program B, or School A to School ^. 

This approach can be labeled the Traditional Model of 
Evaluation. It is usually implemented in the following manner: 
an outside expert (consultant) is hired to do an evaluation. He 
looks around for a few days to get a "feel" for the enterprise, 
selects a set of standardized tests that He thinks have something 
to do with the enterprise and administers them, both pre- and 
post-. The results, showing no significant differences, are 
written up in the form of a critical report. Finally, the 
"evaluator*' collects his fee and possibly publishes the report. 

The major purpose of this model seems to be the professional 
development of the evaluator. Thus another possible title for 
this model is the Evaluator Model, or the Eval uator-as-Expert 
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This is not a legitimate function of, nor purpose for 
evaluation. Furthermore, it is not even a sound procedure for 
conducting an evaluation, e.g,, simply pre- and post-testing. 
Although 'evaluation'^ and 'testing'* have usually been used inter- 
changeably in educational research, evaluation is more than just 
testing:. 

This conception - Evaluator-as-Expert-Model - of evaluation 
is both narrow and usually not very useful to the decision-make??s 
for Khom it is done. In terms of the decision-makers involved,' 
these types of evaluations provide little if any useful data on 
v;hich to make decisions regarding program strengths and weaknesses, 
redefinition and refining of program processes, etc. This in 
fact explains to some extent why so many seemingly excellent 
evaluations (excellent at least from the perspective of the 
evaluator) have been written, bound and put on the shelf, there 
to remain unopened and unread, their conclusions and recommenda- 
tions being ignored, not acted upon. And the evaluator who 
conducted the study can^t understand why such an 'excellent" 
report is bein^^ ignored by the project's decision-makers. He 
fails to realize that the data which are not bein.q used by the 
decision-makers must not be relevant to then anc^ their needs, 
and that this factor is due in part to his own narrow conception 
of evaluation. 

The function of evaluation must be to provide relevant 
data to some decision-makers with respect to some project, i.e., 
data they will use for decision mak5.ng. This is, it will be 
noticed, a much more useful concept of evaluation than the 
pre-post test approach and administering of a standardized test 
at the end of the year. 

Another traditional approach to evaluation has been to have 
a Board of Experts come into an enterprise to do the "evaluation". 
This is found in its highest form in the Accreditation Model, 
with which most school personnel are familiar. The Accreditation 
Team looks at the physical plant, number of chairs, number of 
books, etc. It doesn't really look at program outcomes. Such 
reports are usually very descriptive about very "physical" things . 

O 
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Quality of learning, seldom enters the picture. The real concerns 
of the enterprise's decision-makers are not the focus of such 
"evaluations" , 

However, moving away from these traditional concepts of 
evaluation, it is not only possible but essential to discuss a 
more effective and useful concept of evaluation. As Stufflebeam 
has written. 

Evaluation is a science o^ relating antecedent con- 
ditions and processes to outcomes and outcoTnes to 
objectives. Evaluation strives (1) to determine 
the extent to which objectives are achieved - to 
neasure and define outcomes, and (?) to uncover the 
functional relationships bet we en outcome and procet^s 
variables - to explain outcomes, ( 1967a, p, .12/') 
This definition is not necessarily inconsistent with the pre- 
post test approach. However, it has to be :aken in conjunction 
with another concept, namely, tliat of " -ecision-naker ' and 
'decision making''. 

Decision Maker and Decision Makin^:^ 

This concept is a relatively new one in the history of 
educational evaluation. In 1963 Cronbach offered a new and some-- 
what more comprehensive definition. He defined evaluation broadly 
"• • • the collection and use of information to make decisions 

about an educational program'' (Cronbach, 1963 , p. 672). This 
began a new movement in the field of educational evaluation. 

Since that article, others have taken uo and expanded upon 
this notion, producing most notably the CIPP Model of Evaluation, 
originated by Stufflebeam and Cuba ( Stuff lebeam , 1967a, 1967b, 
1969). This definition of evaluation is typified in the following 
Project operations or activities are evaluated to 
influence decisions which influence program ooerations 
which are in turn evaluated, ad infinitum (Cuba & Stuffle- 
beam, 1968, p. 20). 
Stufflebeam (1969) also writes: 

• • . evaluation means the provision of information 
through formal means, such as criteria, measurement. 
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and statistics, to orovicie rational bases for making, 
judgments which are inherent in decision situations 
(p. 53) . 

These viewpoints are representative of those in the 
literature dealing with the relatively new notion of educational 
evaluation as being decision-raaker oriented. Taken together, 
they represent what can be called a Decision-Maker Model of 
Educational Evaluation • 

Another basic notion needs to be brought up at this point: 
What is, or who are, decision-makers? A decision-maker is that 
person or group of persons who are responsible for making deci- 
sions regarding an educational enterprise* Or, frovs the perspec- 
tive of the evaluator, the decision-inakerC s ) is/are the person(s) 
for when data will\be collected and to whom the collected data 
will be reported for the purpose of assisting or aiding the 
decision naking efforts. 

In the Decision-Maker Model, the actual, in-fact project 
personnel are the decision-raakers and further, their role as 
decision-makers is legitimized in this ModeT. That is, this 
approach to educational evaluation assumes these things, among 
oth ers : 

1) That the pro j ect or enterprise decision-makers , 
be they classroom teachers ^ princioal or super- 
intendent (all of whom are not ential decision- 
makers) have the right • both morally and ethically - 
to make their own decisions abrut their own enter- 
prise. 

2) That it is the responsibility of the project or 
enterprise decision-makerti to make their own 
decisions. It is not the responsibility or the 
right of an outside ''expert^' or "consultant" tc 
do that. 

3) That the only legitimate purpose of educational 
evaluation is to provide information to these 
decision-makers for their own use as they see fit, 

4) That the validity of this approach is determined in 
the final instance by whether and to how great a 
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degree the data are used by the decision-makers in 

making their decisions. 
There are a number of other assumptions that separate this approach 
from the more traditional ones. First, it assumes that decisions 
can be made more effectively with appropriate data. Implicit in 
this purpose is that data, to be appropriate, must come from the 
decision-makers' individual project, not from some external sources; 
and furthermore, that the decision-makers involved must believe in 
and be ready to use the data that are to be collected. Thus, 
evaluation takes on a new relevancy when based on internal needs, 
wants, criteria and data rather than on the outdatedness and 
ineffectiveness of the application of external (and therefore 
probably unrelated) standards and criteria to a project. 

This conception also demands that the decision-makers 
involved have the final say in the determination of what data 
they want and need to make the kinds decisions they deem 
important and necessary, not data defined solely by an evaluator, 
or data determined by arbitrary external criteria. 

It is assumed further that evaluation is not a one-shot, 
post hoc procedure, where if the tests show you have succeeded 
by 90^. you can sit back and relax, patting yourself on the back 
(although not knowing where you succeeded and where there is still 
room jfor improvement) or conversely, if the tests show you failed, 
e.g., achieving only 20%, you ^roan and chalk up a lost year, still 
not knowing where you failed or what parts if any are working. 
To be effective, evaluation must be built into a program from 
the first so that the constant and continuing decisions which 
need to be made during a program can be made on the basis of data 
wherever and whenever possible, rather than on impressions or 
intuition alone. 

Finally, it demands that before any data be collected, the 
decision-makers involved need to know not only what data they want, 
but also what data they need and will use, why they want it and 
how they are going to use it. In other words, they must define 
the goals of their project in order that appropriate data may be 
gathered. Notice here also that this is an internal problem, not 
an external one. 

O 

ERIC 



9 

An evaluator's job within this framework of evaluation is 
to assist the decision-makers in stating goals, in deciding what 
data are to be collected and how they might be collected. An 
evaluator's job is not to dictate which goals are important, which 
goals should be chosen, what is ''good'' or ''bad" and so on. 

This approach to evaluation is essential to decision-makers 
who are concerned with how well they are doing by their own 
standards, where they are failing and so on. This approach does 
not tell the decision-makers what decisions to make, but rather 
only shows the^ where they need to be made. 
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II. THE FIRST STEP IN EVALUATION 

At this point, some decision-maker in the enterprise makes 
the decision (and follows through on it) to have an evaluation 
done. He contacts an evaluator and sets up an initial neeting. 
What kinds of things should be expected at that first meetin^^? 
What should the decision-maker look for? What should he ask and 
expect to be asked? This section of the paper focuses on these 
questions . 

Negotiation of the Contr act: Initiation o f t he Evaluation 

The purpose of this first meeting between the evaluator and 
the decision-maker who has been responsible for having the meeting 
set up is to develop the scope of the work for the evaluation. 
What kind of decision-maker would organize such a meeting? It 
could be the assistant superintendent who has been asked by a 
group of teachers, or the superintendent or some other decision- 
maker(s) to contact an evaluator. It might be a team leader or 
a principal who feels a need to have an evalu^.tion done and so 
proceeds to contact an evaluator. In short, it could be any 
decision-maker who has some lepal and financial ability to bring 
in an outside person to do work, in this case evaluation work. 

Assume now the evaluator has come to a meetine with the 
project or school or enterprise. What happens? The decision- 
maker should expect to be asked the same question posed a few 
pages earlier in this work: ''Why do you want to evaluate?'' The 
purpose of asking this is to make sure that it really is evalua- 
tion that is needed and wanted and not something else. If the 
purpose is to provide some kind of data for decision making, then 
the majority of educational evaluators practicing evaluation today 
will probably continue the discussion. If some other purpose is 
given, then the evaluator might (probably) try to help the 
decision-maker specifically define that purpose and then suggest 
another type of consultant who might better help achieve that 
purpose (e,g., a PR man or a needs analyst). 

Following agreement on the purpose of evaluation, the next 
likel/ thing to happen is for the evaluator to begin to explain 
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what he or she can and can't do in tern-s of an evaluation. The 
dccioion makci- aL this point should look for what tasks will be 
accomplished, by whom, and so on. The decision-maker shoul;^ feel 
free to ask any questions that might be bothering hin and clear 
up any confusion he feels. 

If at this point both the decision-maker and the evaluator 
feel comfortable with their respective positions, then the dis- 
cussion will get more specific, or at least it should p:et more 
specific. The decision-maker should expect to be asked something 
like What it is that you want evaluated?'' The evaluator might 
also be concerned with what the purpose of the enterprise is* how 
complex it is, i.e., are there nany parts and decision-makers 
involved or is the enterprise small enough to be viev/ed as a 
single project or program? If the evaluator feels that the enter- 
prise is too broad or too vaguely defined, he will probably try 
to help the decision-maker narrow it down. 

For example, an assistant suoerintendent has invited an 
evaluator to an initial meeting. He says: 

"I want my school system evaluated.'' 
The evaluator sees this description of the enterprise as somewhat 
broad and responds: 

"You want the whole t hing evaluated?" 
The decision-maker responds: 

"'^ell not the whole thin>^, but the reading program." 
Again, to make sure this is the enterprise to be evaluated, the 
evaluator might ask: 

'The whole reading progran, sys tern- wide ? '* 
Not really, just this new reading curriculum we have in 
the Mocel Elementary School.*' 
In other words, the evaluator wants a fairly explicit description 
of the enterprise. He would probably go on to ask what arc some 
of the major elements of the program; some of the major concerns, 
etc. He might ask for a brief description in writing. The de- 
cicion-maker should expect such a discussion. 

This initial meeting will also deal with resources > It 
takes resources to do an evaluation. Resources are defined as: 
staff time, secretarial and clerical support, duplication costs. 
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decision-maker time, and money. In other words, people usually 
think of resources as a fancy name for ''money' but monev is only 
part of resources. The decision-maker should expect then to have 
to identify the resources which will be made available to the 
evaluation. Again, this is ;;oing to probably be more than iust 
quotinr a dollar ($) figure. If the evaluator s3oes not ask to 
have resources identified, then the decision-^maker should raise 
such is sue s as : 

Who is going to type up and distribute prof^ress reports? 

Who will pay for the phone calls back and forth? 

Where will meetings take place between the evaluator and the 
staff involved? 

Who will organize and convene these meetinrs? 

Will there be a final report printed (i^ appropriate)? and 
who will do it? in how many copies? 

Who v;ili print data collection instruments? 
These are just a few of the kinds of issues that need to be re- 
solved during this initial meeting with the evaluator and i^ the 
evaluator does not raise these issues then the decision-maker had 
better or he is liable to find a lot of hidden costs appearing 
later on. Before the discussion concludes, then, the decision- 
maker and evaluator should agree on a list of resources, including 
all those things mentioned above in addition to money. 

Another and perhaps more important issue which should be 
raised and resolved in this initial meeting (and which is often 
overlooked in many evaluations) is to identify for whom the evalua- 
tion is to be done. An evaluation can not be done outside of a 
particular context; in the absence of specific people. An evalua- 
tion is done for people who have particular needs for the informa- 
tion to be collected by the evaluation. (After all, the purpose 
of an evaluation to to provide information to someone for that 
person, or group of people, to use ir\ maklnp decisions.) In other 
words, who are the decision-makers of this enterprise who will be 
provided with data? At first glance, this question may seem simple 
and obvious: "Well, I called you Mr. Evaluator to come here so I 
am the decision-maker." Right? Hot quite. The evaluator should 
respond with something like "Well, do you make decisions about the 
program we are going to evaluate?" 
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Well of course . ' 
'Are you the only one?" 

"No,-" the decision-maker responds, ''there are the teachers 
in the propram who make daily decisions.' 
Is that all?" 

"No, the principal also makes some decisions about it. For 
that matter so does the superintendent. If you start to think 
about it, there are a lot of people who make decisions about our 
reading program. 

As it turns out, for any educational enterprise, be it as 
small as a sin^^le class or as large as all Title III projects in 
the country, there are many, many decision-makers and not just 
those usually thought of as decision-makers (e.r., adminis trators ) • 
For example, in an evaluation done of an experimental K-1, inte- 
grated day Title III project, decision-makers identified were: 
(1) the team teaching in the program (U persons); (2) the principal; 
(3) the other teachers in the school; (4) the Superintendent: (5) 
the school committee; (6) the parents of the children enrolled in 
the program': (?) the Title III office in Boston, Each of these 
different decision-makers wants and needs different information to 
make their decisions since each makes different decisions from the 
others. To collect different sets of data or information for each 
decision-maker in the above example would cost a fortune I because 
each would require a different evaluation design. Thus it is not 
only important to identify decision-makers, but also to put them 
in some priority order since in all probability it will be impossible 
to pay to have an evaluation done for each of them and a single 
evaluation will not be appropriate for all of them at the same time. 

Part of this discussion then should also provide for prior- 
itizing decision-makers. There are any number of ways this can be 
accomplished but what is important is that it be made very clear 
to all parties at this initial evaluation meeting who will be 
getting in formation . 

A related topic is how much of the resources which have been 
identified earlier in this discussion will be allocated to each 
decision-maker. That is , of the total amount of resources , how 
much will go to the evaluation for the first priority decision- 
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maker(s), to the second and so on. For the example p;iven of the 
experimental K-1 propram mentioned above, 100% of the resources 
were allocated to the highest priority de ci s i on-naker , the K-1 
teaching team. It was decided, however, to report information 
collected for them to the other decision-makers but not to do an 
evaluation for the others. Resources just did not allow for such 
a wide ranged approach. 

(It should be noted that providing data collected for the 
prinary decision-maker to other decision-makers in the enterprise 
does not constitute an evaluation for those "others'. Such data 
may or may not be relevant to these '^others' * decision making needs 
and there would be no way of knowin.t^ if such data were to be really 
used by these others in their decision making. Thus^ simply re- 
porting data gathered for one specific decision-maker to other 
decision-makers within the enterprise is not evaluation" for 
those other decision-makers.) 

Remember, an evaluation can not be all thinrs to all people. 
It has to be determined, at this meeting, what it w i 11 be (and 
do) and for whom. 

This should just about cover what will (should) hanpen at an 
initial meeting between a decision-maker and the evaluator. 
Again, any doubts a decision-maker has should be expressed and 
dealt with' any misunderst andinp:s should be cleared up at this 
meeting; both the decision-maker and the evaluator should feel 
comfortable with eac.^ other and with what each wants to do and can 
do. 
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Review t Of the initial meeting between an evaluator and decision- 
maker 

(1) Have you discussed the purpose of the evaluation and 
come to a tnutual understanding with each other? 

(2) Have you specifically defined the 'enterprise* to be 
evaluated and come to a mutual understanding with each 
other? 

(3) Have you had all your questions answered satisfactorily? 

(4) Have you identified a list of resources which includes 
inore than simply money, but staff time, secretarial 
support , materials, etc.? 

(5) Have you identified the potential decision-makers of 
the enterprise identified in #2? 

(6) Have you ordered these decision-makers as to whom 
evaluation data is to be provided? 

(7) Have you decided what percentage of resources should 
be alloc at ed to each deci s ion -maker ? 

(8) Is the scope of work and responsibility of the evaluator 
and decision-maker (or makers if there are more than one) 
been clearly established? 

(9) Has the time period for the evaluation been clearly 
established? 

Each of these points or questions should be dealt with at an 
initial meeting between evaluator and decis ion-mak er . This re- 
view section can be used by a decision maker, if he or she likes ^ 
to check or assess progress during the first meetin^^. In other 
words, this list can serve as a list of criteria for assessinrt 
what has or hasn't', does or doesn^t occur during an initial meet- 
ing. A decision-maker can know what he has gotten and not gotten 
and act accordingly. 
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Preparation of the Evaluation Contract 

The contract should be prepared followinp^ the initial meeting, 
ju3t described above. It should include all the information used 
in answering the questions above, not just an agreement to do an 
evaluation. It should include the purpose, enterprise, resou'^cos. 
decision-makers, time lines, responsibilities- in short, all the 
topics agreed upon between the decision-maker and evaluator at 
that first meetinp^. 

Once the contract has been prepared, it should be p;one over 
carefully by both parties and both parties should ag.ree and be 
comfortable with each point in the contract or the contract should 
be changed. 

Concluding remarks to the d e ci s i on - maker : 

(1) Unless you are very satisfied with the contract and 
happy with its provisions, don't sign it until you are. 
Otherwise, you may have cause for regret. 

(2) Don't accept the contract that simoly says Mr. Evaluator 
and Model School agree to an evaluation for $X.XX. 
"Evaluation" is a fuzzy concept and can include (and 
exclude) many things. Before you sir i a contract, be 
sure you know what you are getting and that you want, 
need and like what you will get. Simoly, who will do 
what. There are responsibilities required from each 
party involved in the evaluation process. 
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III. A GOALS PROCESS 

Whenever an evaluation is done, it should have as one of its 
steps some kind of goals process. The purpose of such a goals 
process is to identify those intents, or aspirations, or goals 
which the enterprise being evaluated is to accomplish. If the 
evaluation is to collect data, on what is it to collect data? 
The answer to this question is: on those goals the enterprise is 
to accomplish . 

The goals process is a very important part of evaluation. 
It provides for the selection of variables as well as providing 
the basis for designing the entire evaluation. If the Hoals 
process is incorrectly applied, then data to be collected later 
will be less complete, less efficient and less focused than it 
should be. These three factors in turn will cause the evaluation 
to be less effective than it should be. In short, there can be 
no efficient evaluation without a systematic, reliable goals iden- 
tification and priorization process. 

Goals occur on all levels of soecificity and io not have 
attached to them the rigorous criteria of specificity prescribed 
for behavioral objectives by Popham and Baker (1970), or Mager 
(1962). Table I lists some of the possible differences between 
the two classes of phenomena. Goals embody intents , the intents 
of the decision-maker, not just the verbalized, specific statement 
of what the decision-maker thinks his behavioral objectives are. 

Because of recent trends in education, it is important to 
clarify terminology. For example, it is important to distinguish 
between the concepts of ''goals* and ''objectives'- This is a 
crucial distinction to understand. The use of the word ''goal'' is 
intentional. The popular catchword in education today i« "be- 
havioral' or "instructional'' objective. However, there is a dis^ 
tinct difference between the "goal'' concept and the "'objective^' 
concept, which is, or should be, a subset of the goal concept. 
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TABLE I 

SOME POSSIBLE DIFFERENCES HET^rEEM GOALS AND 
BEHAVIORAL OBJECTIVES 



GOALS 



BEHAVIORAL OBJ^CTIVrg 



General, vague, not very 
specific . 

Fwzzy nay overlao with other 
rzoals : may be in conflict 
with other goals. 

Embodies real intents. 



4. Does not really connunicate 
specifics to others. 

5. May be stated in terms of 
anybody, including inanimate 
ob j ects . 

Cxamo les : 



1. Specific behavioral verb. 

2. Siny^le speci'^ic verb obnect, 
excludinR nossibilitv o"^ 
overlao . 

3. Reflects writer^s ability 

to write behavioral obiectives. 

4. Communicates very well and 
specifically to others. 



Statec3 in terms 
learner . 



ff the 



1- to have individualized 
instruction . 

2. self-actualization 

3. autonomous learner 
U. open classroom 



The student must be able to 
correctly solve at least 
seven simole linear equations 
within a period of thirty 
minutes . 

Given a human skeleton, the 
student must be able to 
correctly i^^entify by label- 
ing at least UO of the 
followinf^ bones*, there will 
be no penalty for Ruessin^; 
(list of bones inserted 
here ) . 

The student nus t be able to 
spe 11 correctly at le ast 
80% of the words called out 
to him during an exam in at ion 
period . 



(These are taken from T'a.f^er , 
1962 , np. U5-50. ) 



Rather than asking the decision-r.aker to write down c?ll his 
behavioral objectives, as nany traditional'* approaches to eval- 
uation would ask, following which the evaluator woiilu then proceed 
to "measure'' their achievement, a different tack is called for. 
This different tack is necessary for several reasons. First, the 
former approach assumes certain behaviors, skills and knowledges 
on the part of the decis ion-niaker : (1) the ability to write be- 
havioral objectives; (?) the ability to translate the decision- 
maker's purposes or intents into mean ingf ul behavioral obiectives; 
(3) the ability to write objectives embodying all his intents r 
To assume these skills on the part of any decis ion-F>aker is both 
illogical and potentially damaging to the overall evaluative 
effort. (For further discussion of this sub-^ect, refer to 
Hutchinson and Benedict, 1970; Benedict, 1970). 

The decision-nakar is asked what he would like his "enterprise ' 
to accomplish, the word 'enterprise*' being defined as that entity 
about which data is to be collected. (An enterprise can be a 
school, project, class, program: that which is to be evaluated). 

This approach, using an interactive relationship between 
decision-maker and evaluate- should yield an initial list of 
'goals . The most noticeable quality of this initial list is that 
these 'goals^' are usually vague or nebulous. Differentiated 
staffing; educate good citizens; graduate responsible Americans: 
all of these might be typical of the level of specificity of goals 
at this initial level. Even though they are stated as fuzzy con- 
cepts, they embody real intents and aspirations on the part of 
the decision-maker. 

It should be pointed out that fuzziness is not always "bad". 
It is ''good * in the sense that it serves the purpose of allowing 
people to operate in the ordinary conmun i cat i on process of the 
day-to-day world. People coinmunicate in fuzzy concepts; they 
dream in terms of fuzzy concepts and they aspire in terms of fuzzy 
concepts. If these fuzzy concepts are avoided by going immediately 
to behavioral objectives there is the great risk that the be- 
havioral objectives that are identified will not add up to the 
full set of the decision-maker's aspirations. 
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'/hat is important, then is that the elicitatJon of Roals be 
as complete as possible, whatever they may look like Rrammat ically . 
It is essential that the ev&luation be^in with all the goals. 
Otherwise there is the Possibility of missinp: or omitting what 
might be some of the most important intents of the decision-maker 
for the project. (Beginning with goals is possible because a 
methodology does exist for dealing with the fuzzy concepts in 
goals: the Operat ionalization process discussed later in this 
paper) . 

A goals process should have at least three Tnajor provisions: 

(1) mechanism' for generating a list of items or goal stateynents; 

(2) a mechanism for insuring the completeness of the list; and 
finally, (3) a mechanism for ordering (or prioritizing) the list 
of goals. 

Generating a list of goals : The evaluator should elicit the 
decision-maker's goals, being very careful not to insert into the 
process his (i.e.» the evaluator's) own goals, nor his own inter- 
pretation of the decision-maker's goals. Beware the evaluator who 
debates a decision-maker's goals with him: who tells the decision- 
maker what he (the evaluator) thinks the decis ion-makp' s goals 
should be. If the evaluator "forces'' a goal on the decision- 
maker which the latter really does not want or does not hold, then 
data collected on that goal will not, and cannot, be used for 
decision making and the evaluation will either be incomplete or 
fail entirely depending upon the extent to which this ''forcing'* 
occurs . 

Insuring c ompleteness of the goal s list : As pointed out 
earlier, one of the purposes of a goals process is to arrive at 
as complete a list as possib le of decision -maker intents . The 
test of completeness mechanism helps to achieve this purpose* 

One of the criteria of evaluation is that the data provided 
be complete", and the notion behind a test of completeness stems 
from this concept of "completeness" in evaluat\on itself. Com- 
pleteness in evaluation means that (within the resources available) 
all the data a decision-maker needs to make his decisions is pro- 
vided to him by the evaluation. To insure this, at each of many 
decision points throughout the evaluation it is necessary to 

ERIC 



21 

*'test the completeness' of many different processes. By doing 
this throughout the evaluation, rather than at say a terminal 
point, the evaluation design becomes more complete; data provided 
to the decision-maker will also be more conolete. 

The thinking behind how a test of completeness works is 
basically this, A decision-maker, in being asked to think o^ a 
certain class or set of phenomena, may spend an hour or two doings: 
just that. However, this causes him to have a certain psychologi- 
cal set about those phenomena, or, he becomes 'locked into*' a 
certain pattern of thinking. To ask him to keep thinking in this 
same pattern is not useful for he has probably exhausted the 
process from that perspective. A test of completeness is meant to 
jolt him out of that set or pattern by offering or stimulating^ 
the decision-maker with a different perspective, a different set 
of phenomena, to which he may react. After having him get into 
this new pattern by reacting to a set of phenomena from a differ- 
ent perspective, he would again have a certain psychological set. 
And, depending upon resources at the various points of the evalua- 
tion, he would then be presented with yet another set of phenom- 
ena from a different source and so on. It is very important, then, 
that the evaluation have some provision for insuring completeness 
of goals. Such tests of completeness should not be the evaluator's 
own goals but should come from within the decision-maker's 
enterprise . 

Ordering the goals list : Once the goals list has been gener- 
ated and tested for completeness, it is necessary to put i"^ in some 
sort of order. This list may contain anywhere from one to one 
thousand goals. It is impossible physically (and financially) 
to proceed with an evaluation on twenty or thirty fronts at the 
same time. It is necessary to proceed at one point, A Prior- 
itization mechanism provides for a systematic ordering of the 
decision-maker's goals such that the evaluator will know how to 
proceed. It is very important that the decision-maker decide this 
order (with the evaluator assisting him in an objective and 
systematic foshion). The evaluator should not determine this 
hi::iself . 
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Review : A Goals Process 

When an evaluation is being done^ docs it do or have the 
following : 

(1) use the decision-maker's goals? 

(2) ensure that the goals are really those held by the 
de cis ion- maker? 

(3) ensure that the cvaluator does not interfere by inserting 
his own goals or feelings? 

(4) that as many as possible decision-maker goals are 
identified? 

(5) that there is an ordering process of some kind that 
results in an ordering that is acceptable to the decision- 
maker? 
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Unless resources - including time, staff and noney - are 
extremely limited* an evaluation design should have as one of its 
steps a parts" process. What does this mean? 

One type of evaluation information or data one ofen sees looks 
something like this: 



The evaluation is done near the "end" of the project. 

We might term this a post hoc evaluation procedure where some 
sort of measurement or testing is done at the end of the project. 
This is a one-shot type of evaluation. 

But 5 the next question is "So What?" tfhat usefulness is 
there in deciding that the enterprise is doin^ well or poorly or 
is 63% satisfactory? What decisions can decision-makers make on 
the basis of this? If the report shows 80% success in June, does 
the project pat itself on the back and applaud? What if the re- 
port shows only 20% success? Does the project then wrin^ its 
hand and chalk up a v;hole year to failure? Furthermore, what was 
80% or 20% successful anyway? 

In short, such information is of little utility in knowing 
what succeeded or failed. The utility of evaluation should be 
in knowing what parts or components or elements of the enterprise 
are working well and which are not working very well, and in 
addition, knowing this at the time it is happening when there is 
time to correct it, rather than after it is all over. 

One needs to be able to assess each part or component as it 
contributes or fails to contribute to the purposes (goala) of 
tne enterprise . 

Instead of looking at the enterprise as a whole, 



PROJECT 



E 
V 
A 



REPORT : 



63"^ SUCCESS 




ENTERPRISE 



2U 

We look at the components or parts or systems of the enterprise, 

enterprise 
system system 
sys t em sys t eTn 



If one has the parts of the enterprise, one can evaluate each 
part as it contributes to the goals of the enterprise. The pur- 
pose of a parts process is to identify the parts of the enterprise 
from the point of view of the decision-maker for whom data is to 
be collected. 

One can find what isn't working which can provide the basis 
for making change and evaluate the change - thus one is freed to 
innovate because one can really know whether or not the innovation 
is better. 

Again, this is in keeping with the idea of providing continual 
data to decision-makers for the purposes of r^akinp the continual 
decision any project must make* One needn't and shouldn^t wait 
until it is all over and then either shout or cry. 

How might this be done? The evaluator should work with the 
decision maker to identify the parts of the project being evaluated. 

This is not as difficult as it may sound. Every system has 
a certain number of givens, i.e., given elements. Among these 
are Inpuv, Interfaces and Output. 

Input : those things occurring before the enterprise begins , 
or those pre-requis ites for the program. Examnles 
in a school situation might be budget, a physical 
plant and so on . 
Interfaces: those things which are not directly a part of 
the project but which impinge on it and thus in- 
fluence it. Examples again in a school situation 
might be the School Committee, parents, PTA, Legis- 
lature and so on. 
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Output: that which results from the project or program, 

that occurs after the program is ended. In a school^ 
the output might be the student after the program 
or at the end of the year. 

Now, for evaluation purposes, what is needed is the decision- 
maker's conceptualization of what these systems of the enterprise 
are* The decision-maker should be asked to list the major con- 
ceptual components or parts of the enterprise. For example, 
•'When you think of your enterprise, what are the major things 
(parts) in which terms you think of it?'' 

The evaluator should not tell the decision-maker what his 
(the decision-maker's) parts or systems are. He may tell the 
decision-maker about Input, Interfaces and Output as s;€neral 
categories, but the evaluator again should not fill in the content 
of the categories for the decision-maker. The evaluator should 
also not give the decision-maker too many examples because the 
evaluation design might end up with someone else's, not the 
decision-maker ^ s , components. If this were to happen, the eval- 
uation will begin to lose its efficiency. 

Several other points should be made hex>e about a "parts^* 
process. Different decision-makers may and do conceive of the 
same enterprise (or system) in different ways. (Example I) shows 
components of a school of education from the perspective of the 
Dean (a decision-maker in such an enterprise). (Example II) 
shows the components of the same school of education from the 
perspective of a School Council (another decision-maker in the 
same enterprise). These two examples show how a single enterprise 
can be viewed very differently by different decision-makers with- 
in it . A third example (Example III) is also provided, which 
shows the components of an Early Childhood Program from the per- 
spective of the teaching team (the primary decision-maker in this 
particular enterprise) . 

In the three examples given, the enterprise has been broken 
down one level. Conceive of the enterprise as a whole as level 0 
of breakdown. Once the major parts of this have been identified, 
consider these the first level of breakdown. Each of the systems 
at the first level of breakdown are in themselves systems. As 
such they have input, output and interfaces, and other subsystems. 
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The next step in a parts nrocess is to go to the second level 
of breakdown for each of the systems identified at the first level 
of breakdown. 

For example, look at the system labeled 'Climate" in example 
IIIA. Climate is the first level of breakdown fror Example III, 
In this instance, when broken down one more level, i.e., the second 
level of breakdown, two sybsystems were identified: 'Physical 
Climate' and ^'Affective Climate,'' 

An evaluation design should provide then for some kind of 
'parts" process, from the perspective of the decision-maker for 
whom cata are to be gathered. The parts process, like the F:oals 
process, should have at least three major provisions: 1) a 
mechanism for identifying (or generatinn) an initial list (or set) 
of parts; 2) a mechanism to insure that all the major parts have 
been identified; and finally, 3) a mechanism for matching goals 
to parts since the original purpose of parts was to be able to 
evaluate the enterprise in terms of its parts vis-a^vis goals, not 
the vihole enterprise. 

The purpose of the first mechanism and what it might look 
like are described in the beginning part of this section. In 
terms of the second mechanism, as with goals, the objective here 
is as complete a systems breakdown as possible. The more complete 
and specific the analysis of systems, the more specific and mean- 
ingful data can be related to specific parts of the Project and 
not the project in its most global sense. 

Concluding Remark s_ : 

Do NOT be alarmed, or frustrated, or depressed and throw up 
your arms and say "I'll never be able to do all this", You're not 
supposed to - the evaluator is. This material is being presented 
here so that when you hire^ an evaluator, you will know the kinds of 
things to look for, to expect and the purpose of these processes. 
This material is also being presented here so you will have some 
cr"^teria against which to measure, or gauge, or evaluate'* the 
evaluator and the evaluation. 

Is evaluation complex? Yes, it is. 

Is it easy? No, it is not. 
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This material it is hoped., will better allow you to go into 
an evaluation with your eyes open, knowing what to look for, a 
little less anxious than you might have boen. Evaluation is 
meant to help you and if it doesn't, then it, the evaluation, is 
not working, and needs to be irrproved. You are the decision-maker; 
the evaluator is the evaluation expert. 
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Review : A Parts Process 

Then let's review this section as to what to look for in an 
evaluation : 

(1) Does it have or make provision for oroviding data in 
terms of parts of the enterprise? 

(2) Do the parts come from the decision^maker for whom data 
is going to be collected? (They should.) 

(3) Are there nechanisms for generating a list of parts? 
for insuring the completeness of the parts list (or 
diagram if you prefer)? for match inr the goals to the 
parts? (There should be.) 
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V. A :!atchi::g process for goals a^d part'^ 

Once the goals have been identified from the goals process, 
and the parts have been identified fro"i the parts process, there 
is a need for a process to relate goals and parts to each other. 
A prioritized list of goals has (or should have) resulted from 
the goals process and a prioritized list of parts should have re- 
sulte-i from the parts process. Now, these need to be matched to 
each ether. This is done because of the purpose of doino: a parts 
analysis in the first place; to increase the efficiency and use- 
fulness of the data ^>?hich is to be provided ^or decision making. 

One way of doing this matchinc; job is shown in the ex=imple 
diagramiT^ed on the next pare. The enterprise in this particular 
evaluation is a high school course in matheT.atics and the decision- 
maker in this particular instance is the teacher of that class. 
The goals, listed in the left olumn were h_is (the teacher's) 
goals for the enterprise and the parts on the top row were also his. 

Vihevevev an X appears in a box, it indicates that the goal 
in the column is supposed to be accomplished, at least to a de- 
gree, by that part, or system, of the enterprise. Each and every 
goal should relate to at least one part. Each part should have at 
l^ast one goal related to it. Such a diagram makes it possible to 
observe if there are goals for which no part has been identified 
to fulfill them; (Is there a goal and no'X's" in the row next to 
it?) This example does not provide an instance of this occurring 
but should it occur, it would indicate a need to the decision- 
maker relative to the design of the enterprise. 

Such a diagram also makes it possible to see if there are 
parts without any (seemingly) useful function. (Are there any 
parts under which there appear no '*X*s"?) Again this example does 
not provide an instance of such useless parts but should such have 
appeared, it would have indicated a need to the decision-maker 
relative to the design of the enterprise. 

The Evaluation should not tell the decision-maker to make a 
decision or that a decision is needed. The Evaluation would simply 
provide data and point out any discrepancies (such as the two 
pos*^ible cases described above of missing parts or useless parts) 
and leave any decision making up to the decision-maker. 
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Rev iew : Goals /Parts latching 

(1) Does the evaluation have a provision ^or somehow match- 
ing the goals of the enterprise with the rarts of the 
enterprise for the decision-maker? (It should.) 

(2) Does this Tnatching process use the goals identified 
froTH the goals process and the parts identified froTn 
the parts process for a given decis i on- r^aker ? Or does 
it use one decision-r:aker ' s parts an^'' another's goals? 
(The latter shouldn't happen.) 

(3) Does the r^atchinr: process prcvide for the decision- 
maker doinp. the matchinp? (It should.) Or, is the 
n;etching done by the evaluator? (It shouldn^t be.) 
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VI. AM O^ERATIONALIZATIOM PPOCF.SS'- 

This is one of the most important processes within an eval- 
uation. It deals directly with the problen^ o^ translating^ what a 
decision-maker v/ants to do, into an observable or measurable state. 
It is also an area where such current evaluation models as 
Stuff lebea:T\' s CIPP (Context, Input, Process, Product) Model, 
Provus^ Discrepancy Model and the EPIC Model fall far short o^ 
an ideal and in fact, do not satisfactorily deal with it at all. 

Aftei- all these years, there is still a dichotor.ous trend 
in education with vapors to behavioral objectives- On the one 
hand there is Mager ( 1962 ), Blooni ( 1956 ), PoDham ( 1969 ), and 
Popham and Baker (1970), all of whom represent a school of thought 
which would have us detail in minute, behavioral terns the ob^ 
jectives of whatever it is we are about, or else, they pose, 
we'll never know where we are goinp: or vrhere we h^^ve been. On the 
other hand, tliere is an increasing movement with spokesmen like 
Atkin (1963), Ausabel (1967), Raths (1958) and Eisner (1969) which 
questions the efficacy of the former school, su/ip^estinr that when 
forced to operate along Magerian lines, the essence of what we 
are about may very well be lost, or that the behavioral objective 
approach is limited in its ability to deal with things that are 
really or shoulo be of concern and importance to us, e . p; - , 
affective f:oals . Despite Popham's ( 1968^^ excellent refutation of 
this latter point of view, an uneasiness still remains with us 
about the efficacy and desirability of one or the other of these 
two seemingly polar opposite points of viev- 

These two positions may not be polar opposites. The problem 
may be that our abilities of conceptualizinp are still in too im- 
mature a state to handle the non-Mageri ans versus the Magerians 
points of view simultaneously. The point is: 



- The majority of this section originally appeared in, Hutchinson, 
T. E. and Benedict, L- G., T he Operat ion ali zat ion of Fuzzy 
C oncepts , University of 'Massachusetts, mimeo , September 1970. 
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Evaluators , educators, all human beinrs, have enornous 
difficulties in reporting the sum and sweep of their 
objectives. We all have goals and we consciously and 
unconsciously give priority to some pioals over others. 
But we have few reliable ways to report then to others, 
or even to reveal then to ourselves, (Stake and Denny, 
1069 , pp. 375-376 ) 
This is the crux of the natter. V/e all have goals but getting from 
goals to verbalized or explicit statements of \^7hat these goals mean 
not only to others but to ourselves is the problen. 

For example , it is easy to state, ''The student shall solve f3 
quadratic equations in 5 minutes without the use of any materials 
other than scrap paper and a pencil.' It is easy to communicate 
th is to others with full understanding, as it is an easy task to 
determine whether, if and when this objective is accomplished by 
the learner. However, this is not the case with a whole host of 
other kinds of goals, e.g., affective. The student shall be self- 
actualizing. . . , or 'The student shall value his self," and so 
on. These latter goals are difficult to communicate and understand 
and yet a legitimate argument can and is mad^:: that these are im- 
portant as is solving 5 quadratic equations. Yet, while verbalizing 
these humanistic or affective goals, teachers and educators and 
objective-writers have failed to deal effectively with them pre- 
cisely because their conceptualizing abilities have not been ad- 
vanced enough nor comprehensive enough to do so. 

Hhere is the solution? Can there be one? Is it true that 
without Magerian objectives we can not progress anywhere? Is it 
true, as the non-Magerians state, that putting content or goals 
into Magerian terms destroys that which is to be measured? 

To date our conceptualization strategies have been limited. 
A possible bridge from the Mager to the Atkin position, i.e., a 
possible solution to this dilemma, may have been developed bv 
Hutchinson (1969a, 1969b) ^ perhaps quite accidentally while working 
on solutions to other problems. He may have come up with a process 
whereby both the Magerians and their opposition ui:.l feel not only 
comfortable with what they are doing, but with each other. They 
need not seem to be polar opposites any longer, nor mutually ex- 
clusive, since in reality (it is contended) they are simply dif- 
O ferent points on a single continuum. 
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Examine for a ?nonent sone of the berinninr of thi?=; controversy. 
Why is it that objectives ever bef:;an? It could have started when 
evaluation or assessment of student achievement bejpan. It really 
cane into focus with proprammed learninr with which Maper was 
really concerned wnen he wrote his book. The problem actually 
had its basis in the need for neasurement. And this is the point 
at v/hicri evaluators entered the scene. 

Evaluators and evaluations have had and continue, to have a 
bad nana. They are associated with anxiety on both the teachers' 
and students' parts. They have too often been part of the first 
school of thouf^ht mentioned earlier: "Tell me your specific be- 
havioral objectives and then I will evaluate is typically assip;ned 
as cominp, from an evaluator. As Stake and Denny write ( 1969 ), 
"An evaluator's technical skill should help the educator convey 
his purposes, both those that quickly come to mind an--^ those im- 
plicit in what he does. VJhat are the present methods . . . Our 
methods nov; are crude, unst andardized and unvalidated. They should 
be more evocative, mere sensitive than indicated by the bold re- 
quest, Please state your objectives in the followin^T space." (p. 376 ) 

Hovrever, the above is not the only shortcoming of evaluators. 
A second is that of the subjective approach to evaluation, all too 
common a practice today. In this method of evaluation, the evalua- 
tor enters the situation and 'feels" what is happening, or tries to 
sense some sort of global dimensions of what's happening, a'Pter 
v:hich the evaluation is written. The problems with this approach 
are all too obvious- 

Yet a third dimension which contributes to the fear anc^ an- 
xiety associated vjith evaluations is that the evaluator will use 
outside, unknown or irrelevant criteria to evaluate 'my school'' 
or "my course*' or ='ME*'. That this point has been compromised is 
evidenced, for example, by such criteria for a Social Studies 
Evaluation, as provided in the Natural Study of Secondary School 
Evaluation's, Ev aluative Criteria (I960) as: enrollment^ number 
of sections, range of class size, class periods per v/eek, room 
arrangement and so on» 

These problems v;ith the current state o^ evaluations need not 
be the case. In fact, the whole nature of evaluation, what it is 
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and isn't, what it should and shouldn't do is changing (Stake, 
1967, Stuf flebearn, 1959 , Scriven , 1967 ). Evaluation is headed for 
a new definition for which it indeed is time. 

It is in this new movement of redefinition of the function 
of evaluation, and in developing a much-needed methodology of 
evaluation consistent with this movement that Hutchinson has de- 
vi.sed a procedure he has entitled 'The Operat i onali zat ion of Fuzzy 
Concepts". An initial reaction to such a title is probably 
scepticism followed by ''What is it?" Upon investigating this 
procedure, one discovered an extre^.ely wide r.Tnp;e of potential 
possibilities and applications. One such application is dealinr. 
with educational goals that are not easily turned into behavioral 
ob j ect i ves . 

What is a Fuzzy Concept ? 

Fuzzy concepts are common, V/e all use them everyday our 
lives in communicating: peace, love, democracy, patriotism and 
civil liberties are just a few examples of some of the many, many 
fuzzies used frequently today. Because each of us has different 
perceptions of the same words, such as those above, or phrases 
like sel f- actualizat ion , individualizing instruction and student- 
centered learning, there often arises misunderstanding, disa.eree- 
ment, tension and even conflict. Often one hears the point made 
that what is really at issue is a semantic problem, a communication 
gap. This is due in part to the use of fuzzy concepts. 

Fuzzy concepts can also be said to represent the dichotomy 
between instructional or behavioral objectives and p;oals, or non- 
ins truct ional objectives. This very important difference or dif- 
ferentiation between goal and objective should not be underempha- 
sized, overlooked nor confused. A goal, for example, is an "end'' 
in non -behaviorally defined terms, such as ''The student shall be 
self -act ualizing" . An instructional or behavioral objective on 
the other hand is an operat i on alized goal, e.g., *'The student shall 
list in writing at least 5 directly observable components o-^ his 
self-concept as he perceives it." 

The apparent gap between the tv;o schools of thought on the 
objectives controversy, between 'goals'' and ''behavioral objectives'' , 
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is due in part to the fact that in reality these renrcsent two 
different points on a sinele continuum, not two different continue. 
As Stake and Denny wrote, mentioned above., all of us have goals. 
It is simply a lack of conceptualizing: strater^ies, an absence of 
a means by which to show that this gap i? only an apparent one 
that is the issue in this controversy. 

Hutchinson's technique, the operationalization of fuzzy 
concepts, may be the conceptual tool needed to resolve the issue. 
Keepin<3 in mind the definition of f^oals , this nipht be represented 
as shown in Figure I. 
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A goal, when the operationalization technique is aopliod, wtII 
probably yield many behavioral statements (or obiective^). It 
is important therefore not to dismiss goals, just as it is im- 
portant not to dismiss objectives. The premise here is still the 
use of objectives, or operat ionali zed goals. What is important is 
the way or medns by which teachers and other educational decision- 
makers are exposed and introduced to the logic and necessity of 
objectives, as wt;ll as the way in which evaluators go about 
arriving at behavioral objectives. 

Please note: the best way to learn this technique is to 
experience it. In order to maximize this experience the reader 
is asked to practice each step of the procedure as it is intro- 
duced and discussed. To simply read through this section trying 
to do each step will not be very effective for the reader. 



The Operat ionalization of Fuzzy Concepts: A Methodology 

Step 1: The first step in this orocedure is for you to 
choose the fuzzy concept to be ope r at i ona 1 i ze d . 
Some examples are: peace, love, helping others, 
job satisfaction, sel^-fulfillment, etc. The 
reader should choose a fuzzy concent that he uses, 
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or intends to use, rather than or.e which is not 
important or 'neaninRful to nin. For purposes of 
this paper perhaps it v;ould ho easier if the concept 
'helping others" is used. ''rite the ^xxzzy concept 
on a piece of paper. 

Steo 2: Create in your mind a hypothetical situation. This 
hypothetical situation vrill have a group of people 
in it, an environnent, things, furniture, etc. It 
nay be indoors or outdoors. Now, imagine that the 
fuzzy concept exists in this situation and is in 
the epitome, is absolutely 100% present. Ob serve 
that situation and all the thin?;s you see about it 
that indicate to y o u that your fuzzy concept is 
present in this situation. The hypothetical situa- 
tions should be as conolete and real as possible. 
For example, the hvoothetical situation in this 
case night be a ciassroon with chairs, tables, 
blackboard, etc. There is a teacher present, a 
group of students and so on. The teacher^s be- 
havior is the epitOTTie of "helpinq others". List 
those things you can observe in this situation 
that indicate to you that the fuzzy concept is 
present. Some things night be: 

a, concerned with the student as an individual 

b. warm 

c . s i nc er e 

d. considerate of students' opinions, values, etc, 

e . smi les a lot 

f. provides a supportive climate 

g. provides success experiences for students 

h. provides experiences for students to reduce 
the ir anxiety 

i. provides experiences ^or students to define 
and reach their own goals 

Obviously there are many others. Possibly none 
these would appear on your list of your concept of 
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"helping others'. Now, you should write your list 
down. Use this hypothetical situation completely, 
try to identify all the elements of "helping others" • 

Step 3: Now again construct a hypothetical situation and 
again with the environment and furniture, thir.^s^ 
etc- , a group of people and there is present in 
this situation the complete absence of the fuzzy 
concept, e.^., absolutely no ''helping others^' 
present. Uhat things do you see in this situation 
that indicate to you that your fuzzy concept is 
completely absent from this situation. Let's, 
take again the same hypothetical situation as was 
set up in Step 2: a classroom, a teacher, a group 
of students, etc. This time, imagine that this 
teacher is directly opposite the ideal .of helping 
others. List those things you can see in this 
situation which definitely indicate to you this 
teacher is not '^helpin;^ others'. Some examples 
might include : 

a. ignores students* opinions and values 

b. not aware of students as individuals 

c . egocentric 

d. selfish 

e. does not allow for individualization 

f. authoritarian 

g. discourteous 

h. undermines students' feelings, morale, etc. 
Obviously, again, these are only a few possibilities. 
Again, maybe none of these will appear on your list 

^^"^ y o^r conception of "helping others'*. W r i t e 
down all those things in this situation that you 
observe that indicate to you the fuzzy conceptis 
absent. Don't bother with the negative statement 
of the positive elements listed in the previous 
step. Concentrate on identifying those aspects 
that were not already found. 
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Step U: After having gone through both the Dositive and 
nepative hypothetical situations, the chance o^ 
easily finding more dimensions out o^ one's mind 
is not very great. So next we employed some 
strategies called tests of completeness . (first 
test of completeness): Get someone else to .f^o 
through the same steps as above with the same fuzzy 
concept. One then looks at the other person's 
list and considers item by iter, if the item should 
be on one*s own list and if it is, add it to the 
list. Should you decide the item is inapprooriate, 
reject it, i.e., it does not fit your conception. 
Or a third possibility is that the other individual's 
item may make you think of one or more d i mens ions 
you have forgotten (recommended perhaps because you 
dislike their di mension.) Ideally this test of 
completeness should be done with three or four 
other people . 

Write down the appropriate dimensions which result 
from above. 

Step 5: (second test of completeness): Go back and re- 
create the hypothetical situations. Now. there 
were things that you saw in those hypothetical 
situations that you wrote down , i.e., your two 
lists. There were other things that you saw that 
you did not write down. Go back, look again at 
those things that you saw and did not write down ^ 
and seriously consider the implications of these 
not being dimensions. 

To use an example out of the context of ^*helpin^2; 
others", consider fuzzy concept "job sat isf act ion*' • 
If a person were operationalizin g "success in a job", 
one of the dimensions which he reiectec'' in the 
first hypothetical situation mip,ht be money. Now 
the question should be asked, "What are the impli- 
cations for success in a job where the job provides 
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no money at all?'' Suddenly it becomes obvious 
that for almost everyone money must Play some 
role however slight in job satisfaction- So the 
dimension money is added, but perhaps a qualified 
amount, e.g., $10,000, 

Now consider those dimensions you rejected for your 
fuzzy concept and write them down on your list if 
on reconsideration they are for you, a part of the 
concept • 

Step 6: (third and last test of completeness): The task 
here is to deliberately construct some dimensions 
th at have nothing to do with your f uz zy concept , 
in this case 'helping others' , and again, consider 
the implications of these dimens ions for your 
concept. Try that and in fact, write them down. 
Start out by askinc; yourself, "What has nothinp, to 

with (fuzzy concept)'^ and then, ''Does it 

really matter?^' 

The example of our teacher "helping others" provided 
us with a number* of dimensions of this concept. 
Mowj did you consider' the teacher's family life? 
relationship with his or her peers, the administra- 
tion? Probably not, but is it not possible that 
each of these could have serious implications on 
that teacher's "helping others'*. The purpose ho^^ 
is not in fact to find things that have nothing 
to do with your concept but rather to attack the 
problem from a different perspective. 
As you proceed through these steps, each one will 
be more difficult as the dimensions that comprise 
your conceptualization of what you mean by your 
f uz zy concept become more and more complete the 
number not identified become fewer and fewer and 
therefore hard to find. 

After one has gone through the 6 steps in sequence, 
it is reasonable to conclude that one has a fairly 
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complete list of the parts of the concept at the 
just level of breakdov/n* This product of this 
process, then, might be represented in Figure 2. 

Now using our example of helping others, as a result of the 
first U steps, some 17 dimensions of "helping others'^ were arrived 
at. Thus on the first level of Figure 2 there are 17 numbers. 
The next step in the process is: 

Step 7: For each item on your list, in this case 17 per- 
haps added to as a result of the tests of complete- 
ness, the reader should ask himself, *^Can I observe 
that dimension directly?" Somethin^^ which can't be 
observed directly is defined as a fuzzy concept. 
Thus, for each item you must decide if it is still 
fuzzy and if it is, then you must repeat, in the 
same order, the sequence of steps above. 

In this particular example, none of the 17 items are directly 
observable and thus each must be further operationalized at least 
another level. Obviously at this point it becomes clear that this 
can be a very lengthy process. It could take nearly forever to 
do a complete operat ionalizat ion . Thus at this point in the 
process » another technique is used, namely priorit izat ton . 

Since time is a resource and all resources exist in limited 
amounts, the reader must decide how much time he can allot to 
operat ionalization , depending on the reason he began the process. 
As an example, let's assume time is limited to a given amount 
and the operat ionali zer decides only items 1, 2, 12, and 14 can 
be operationalized. He repeats the process for each of these, 
including the important Step 7. Again, if an unmanageable number 
of dimensions are found each of which needs further operational- 
ization, the prioritization at level two may take place, as in 
level one . 

For a very fuzzy concept, what usually happens is that very 
few items at the first level of breakdown will be directly ob- 
servable. As the operat ionalization process is carried further, 
a larger percentage are found to be directly observable. 
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Perhaps it would be appropriate here to use a less fuzzy 
concept, one which can be fully operat ionalized in several levels 
rather than a large number. A iuzzy concept for a college physi- 
cal education teacher might be ''competent weif^ht lifter''. At 
the first level of breakdown, there are two dimensions: Olympic 
lifts and power lifts. Asking? the question, are these measureable 
or observable directly, the answer is "no** and the orocess is 
cont inued . 

At the second level of breakdown, 6 more components are 
found, three from each of the first two: Dress, snatch, clean 
and jerk; and bench press, squat and dead lift. Further opera- 
tionalizing ^'competent , certain attributes are attached to these 
dimensions^ thus the third level of breakdown: 

For a weight lifter with a body weight of 12?"^ pounds or less 

press: 150 lbs. 

snatch: 150 lbs. 

clean and jerk: 200 lbs. 

bench press: 200 lbs. 

squat : 250 lbs . 

dead lift: 450 lbs. 
Each of these can be observed or measured by numerous methods and 
thus no longer fuzzy. The lifts themselves are operationalized 
by the current A.^.U. Weightlifting Handbook . (See diagram on 
next pane . ) 

This was obviously a simplistic fuzzy concept with appeal 
to a limited audience. However, it exhibits how the process can 
and does work. 

This then has been a brief overview of the operationalizat ion 
of fuzzy concepts. It was introduced by two potential applications: 
first, as part of a new methodology of evaluation and second, as 
a method of resolving the objectives controversy* 

An operat ionalizat ion process should do the following: 

1. Deal with the most important goals of the decision- 
maker for whom the evaluation is to provide data. 

2. Take the most important goal and systematically break 

it down into behavioral, measurable dimensions or components. 

3. Once the most important goal has been broken down, it 
will deal with the second most im'^ortant goal and so on • 
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FIGU'.^E THREE 
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U. Once operational! zed , a goal or intent will consist o^ 
a whole list of observable or rnia s^rab le itens as in 
the wei ph 1 1 " t in example. 

5. These observable items should be prioritized by the 
decision-maker ('.:ith the evaluator's help if necessary). 

6. Each iten now becorr.es the behavioral iten ^or which''\ 
measurement for evaluation will be done. In other 
words, each item becomes the focus of developinc^ a 
measurement technique which is then implemented and data 
colle cted . 

The results o^ operationalizat ion , then, form the basis for 
developing measurement tc^chniques. This is the reason for the 
importance of the process. If the operat iona lizat ion does not 
work, then data collection will fall far short an ideal or 
best and may even fail completely. 
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Review: An Opera t i ona 1 i za t Ion Pr o cess 



An evaluation should have some kind of operational- 
ization process. It r.ay not look exactj.y l?ke the one 
described herein. It may look entirely different. But ^ 
there has to be some sort of operat ionalization process. 
This is essential because of the need to break ^^oals or 
intents into measurable, observable, behavioral state- 
ments. Merely starting with '"write behavioral objectives" 
omits m.uch that is important in terms of what the i^ecision- 
maker wants to accomplish. Therefore ^-.n evaluation which 
starts with 'behavioral objectives^ is fallinj^ far short 
of the ' ideal' and the decision-maker should be aware of 
this . 
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VII. MEASUREMENT FOR EVALUATION 

Obviously one of the most important parts of evaluation is 
the collection of data. Data are collected using various ob- 
servational techniques. The decision-maker for whom data are 
to be gathered and reported has a very important interest in the 
techniques which will be used to collect data. Therefore he 
should be involved in the development and/or selection of such 
techniques . 

If the purpose of evaluation is to provide data for decision 
making; and if the data provided are to be u*?ed by the decision- 
maker-, then any techniques used to collect data must be perceived 
as valid by the decision-maker or he will not use the data. 

For example^ if an evaluator is hired and he proposes to use 
a standardized test his concern or company has designed, the 
decis ion-makor should carefully examine it to see if it looks to 
him, the decision-maker, as though the information it will collect 
will be useful, that he will be able to use it* If the decision- 
maker feels that most of the information the instrumant will 
collect will be useless to him - ''It measures things I am not 
doing' - then it should not be used. Rather, a tailor-made in- 
strument or technique should be used. 

Most educators have had, at one time or another^ a course in 
basic testing or in tests and measurements. Two concepts that 
most educators remember are "Validity*' and "Reliability". Prob- 
ably no two measurement concepts have been as referred to, or 
over referred to, in evaluation as these two. 

What is validity? A technique is valid if it accurately 
measures what it intends to measure. For example, using a ruler 
to measure the width of a room is a valid technique. A ruler 
measures what it is supposed to measure: distance. 

There are many kinds of validity but one of the most import 
tant, and the one most frequently overlooked in "evaluation" is 
' vlecis ion- maker validity " . Decision-^maker validity simply means: 
do you , the decision-maker, think that the data collection device 
suggested by the evaluator will collect the data that you want 
collected? that will be of use to you? In other words, do you . 
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the decision-maker 5 perceive the instrument as bein;', valid 
(measuring what you think it is supposed to measure)? If the 
answer to these questions is "Yes**, then the technique or instru- 
ment is said to have decision-maker validity. If you, the deci- 
sion-maker, are skeptical about an instrument or measurement 
technique- or have doubts about its ability to do what you want 
it to do, measure what you want it to measure, then the intrument 
or technique is said to lack decision-maker validity and should 
not b-3 used. 

What is reliabijj.t y? 

Does the technique perform consistently with time? For 
example, if we had a ruler which expanded several inches on a 
hot day or contracted several inches on a cold day, it would not 
be a reliable measurement technique because it would not ner^form 
consistently each and every time it was used. A technique has 
to be reliable (consistent) or it should not be used. 

An instrument can be completely reliable and very 'valid" in 
the traditional testing; sense and yet supply completely irrelevant 
data to the decision-maker for whom it was intended to collect 
data. In the past, traditional tests, testers and evaluators have 
concentrated on 'validity' (not decision-maker validity) and 
reliability to the exclusion of the decision-maker's needs. (This 
is only^ one reason why so many ^'traditional" evaluations have 
failed, i.e., have sat on the shelf and collected dust.) 

In terms of evaluation, when it comes to the measurement, 
the dec is ion -maker should expect some interaction with the 
evaluator on the development and/or selection of a technique. If 
the decision-maker leaves this entirely in the hands of the 
evaluator, chances are very good to excellent that the data 
collected v/ill not be completely useful for making decisions and 
possibly will be entirely useless to the decision-maker. There is 
the example of the outside evaluator hired to come in and evaluate 
p summer workshop v^hose purpose was to take pre-school, disadvan- 
taged children and give them readiness activities in preparation 
for their entry into first grade. The evaluator arrived with two 
tes's in hand, administered them? wrote up a report ehowinp a few 
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sionificant differences, (mostly no significant differences) and 
sent the report to the decision-makers. The decision-makers re- 
acted: ^Neither test neasured what we were doinp! ' *^We were 
dealing with emotions and attitudes and he (r5r. Evaluator) tested 
coj^nitive development' . 

In this example, both tests had been field tested, v/ere 
valid in testing terms and reliable but did not have decision- 
maker validity. As a result, the de cis ion- makers rejected the 
vJhole evaluation, fired the evaluator cind decided to find an 
evaluator who could develop and provide measurement techniques 
which could collect data about what they (the decision-makei^s) 
were actually doins;. 

In the first place, then, an observational technique must 
fit that which it is to measure. It must be developed or selected 
from existing techniques for a specific task: collecting data 
on a specific goal or intent which the de cis i on - mak er may hold 
for his enterprise. Prepackaged tests or standardized often fail 
to do this since they are usually on such a general level (in 
order to measure a wide range of things) th^t they miss collecting 
data on the specific needs of a specific decision-maker. 

Part of decision-maker validity is determining, by the 
decision-maker, for himself, whether a technique seems to fit 
that which it is to measure. If an instrument is clearly foinr. 
to measure cognitive development and the major concern of the 
decision-maker is psychomotor activity or affective components of 
that cognitive development, then rec^ardless of how valid or 
reliable is that mf»asure of cognitive development, it will fail 
in this instance because it does not measure what it is supposed 
to. It would not have decision-maker validity. 

'But", the decision-maker is going to say. How do I_ (we) know 
about validity?" Sometimes it is just a feeling, an intuitive 
distrust basod on experience, as with the example just given. 
However* there are a number of criteria a decision-maker can use 
to determine whether an observational technique is useful, valid, 
and going to serve his needs. 
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Crit eri a to assess observational tech n i q u e s 

The decision-naker can ask hin^aelf: is the technique 
d irect obse rvat ion of behavior or is it indirect observation . 
Direct observation is always preferred to indirect because it 
gives a much better indication of what is really hapnening. For 
e>!aTr,ple, if the item to be measured is * children fightinr in the 
schools' it would be best to collect information by direct observa- 
tion - counting the number of fights per day - than to fi;ive a 
self -report questionnaire to all the behavior problems in school 
asking them to write down the number of fights in which they have 
been involved. Students and non-students alike know how to 
•'distort' answers on a written test to the direction the question 
asker wants. They know they are not supposed to fight so they 
report ''no fights" when in fact there nay have been several. 
In such situations direct observation is always preferable to 
indirect . 

Is the technique obt rus ive or unobtrusive ? An obtrusive 
measurement technique is something which is not ordinary but which 
is introduced only foi the evaluation'' so to speak. Obtrusive 
techniques share the same problem that indirect measurement had 
above: it interferes with that which is beinp measured and may 
very possibly alter it. For example, if the item to be measured 
is ''cheating" (the peeking kind) an obtrusive technique is to have 
two or three persons stand in the room to watch for peekincr. An 
unobtrusive measure might be to have a one-way mirror and to 
stand V;ehind it and count the nu/nber of peeks. Unobtrusive 
measures are preferred where possible to obtrusive ones. Perhaps 
the best example is the annual or semi-annual trip by an admin- 
istrator to ''evaluate" the teachers. The administrator comes into 
a teacher's room with his checklist or pad of paper, sits glar- 
ingly or even smilingly in the back of the room busily writing. 
The teacher's behavior will automatically change for the duration 
of this obtrusive" measure. Whether the ch'anj^e is for the better 
or worse is not the point: the point is, what is being observed 
is nojt vhat is usually happening because the obtrusive technique 
is interfering and interacting with that which is being measured. 
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A third criteria which can be used in assessing, measurement 
techniques is that of n aturalness . Is the observational tech- 
nique to be used under natural conditions or under unnatural 
conditions (e.g., test)? That administrator was observing his 
teacher under natural conditions - her natural classrooir^ environ- 
ment but he violated one of the other criteria. Thus it is im- 
portant to note that having just one of the criteria may not be 
sufficient. In the case of the teacher, perhaps again, observing 
through a one-way mirror would have been natural. (Granted, very 
few schools have such devices: remember, this is only for illus- 
trative purposes. ) 

There are other examples o^ ''unnatural^ conditions which the 
decision-maker can be on the look out for in reactinp to or assess- 
ing observational techniques: simulations, models, lab situations, 
test-taking conditions. Each of these is unnatural to an extent 
and vrill therefore distort to an extent that which is being 
measured . 

An i deal observational technique then will be reliable and 
valid (especially decision-maker validity) and it will also ful- 
fill three other criteria: direct, unobtrusive and natural. 
But , as with all ideals, it is very seldom met. Meeting all of 
these criteria will be both er.pensive (usually) and sometimes im- 
possible. The ideal observational technique for determining cer- 
tain behaviors of teachers, say, is an invisible man. This is 
obviously impossible although highly desirable in many circum- 
s t ances . 

However, knowing what 1: ideal, the decision-maker can then 
know how far from the ideal a given observational technique is. 
He can use these "criteria" of idealness to measure observational 
techniques the evaluator presents or develops. It becomes very 
useful, therefore, for a de cis ion -maker to have a rough idea in 
his mind of what an ideal technique might look like for any criven 
item to be measured. 

These criteria become very important in the realm of the af- 
fective domain, psychomotor domain and in the areas of attitudes 
and emotions. In the cognitive domain, there has to be a strong 
rel^'^^nce on paper and pencil tests (again, remembering though that 
even this is far from the ideal) but such 'tests are far from 
j^j^Q satisfactory in the other areas listed. 
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Re view: Measurer^ent for ^vd lua t i on 

(1) Have you, the decision- maker, been involved in the 
development or selection of observational techniques? 

(2) Do the observational techniques have your decision- 
making validity'? (That is do you feel the data 
collected by them can be used by you? Meet your needs?) 

(3) Have they been field tested and been sho;;n tobe reliable? 

(4) How direct is each technique? 

(5) How unob trus 1 ve is each technique? 
(5) How n atural is each technique? 

(7) In short, hov; far from the ideal is each technique and 
is this so far that it loses decision-maker validity? 

Ap.ain, these can be used as criterif< by the decision-maker 
to know what he is getting or is not getting in the way of 
measurement in evaluation. 

Beware ; the evaluator v;ho has one or two or even more pre- 
packaged tests which he plans to administer which you, the de- 
cision-maker have little or no say about. Such tests will probably 
not provide you with useful or useable information and therefore 
should be regarded with skepticism unless it can be shown that 
these are the very best available. (This can be partially 
answered by going through each the above 7 questions with the 
evaluator and posinj^ them to him.) 
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VTII. DATA COLLECTION 

Once an observational technique has been a;rr>eed upon by 
both the decision-iTiaker, who has certified that the technique has 
decision-maker validity, and by the evaluator, who has certified 
that he can use it and that it is usable in terr.s of testing 
validity and applicability, then that technique is implenented and 
data are begun to be gathered. 

There are several criteria which the decision-maker should 
be aware of to use in assessing the process of innleraent inj^ the 
technique. Granted the evaluator (or a measurement consultant 
who night be called in) has expertise in imp ler^ent in observa- 
tional techniques but there are certain things a decision-maker 
can also look at which allow him to make some observations or 
decisions about the implementation of these techniques. 

First, when does the evaluator plan to collect data using a 
given technique? If the evaluator has planned to use a technique 
only once, at or close to the end of the project then the decision- 
maker should question the advisability of this. Data should be 
provided on more than a terminal or after-the-fact basis. The 
decision-maker should use some reference to his needs for data 
before accepting a suggestion to use a technique once, when the 
project is nearly over or the school year is nearly over. 

H ow often should a technique be used? There is no exact 
or correct answer to this question. For example, the following 
is a goal vjhich is held by a teaching team for their enterprise, 
in This case, a primary classroom: 

In the room, many children's thinr;s are displayed. 
The observational technique developed for collecting information 
on this is simply: to randomly pick a tirT?e durin;^ the week; send 
an observer into the classroom to count all thinp.s displayed which 
are children's things (not teacher thinr:s). (Children's things 
include: art, papers, things brought from home to show to the 
c lass , etc.) 

It was decided to implement this technique for the first 
time in October of the year. 
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Time I: In the classrooTn there were 12 children's 

things displayed (drawings, sculpture, papers, etc.) 
The primary decision-maker (the team of 4 teachers) decided that 
this was really not sufficient to meet their intents for this 
goal and so they decided they would work at increasing the 
accomplishment of this intent. In this case, the technique was 
used again a week later and this time. 

Time II: 35 things displayed 
The team decided that they had reached a satisfactory level on 
this and would now turn to other things. 

This does not mean that the technique was never used a^ain. 
It would be used again to see if this level were dropping off, 
staying the same or increasing (each of which would indicate a 
different set of conditions necessitating a different kind of 
decision ) . 

Time III: (4 weeks later): 39 things displayed (all of 
V7hich were incidentally, different from the 35 
things seen U weeks earlier), 
This confirmed the decis ioTt-raakers ' perceptions and feelings that 
this goal was being more than satisfactorily met. In this case 
the technique might not be used again for 2 months. 

But, what if at Time III there had been only 10 or 15 things 
displayed, all of which had been on display when observed U vreeks 
earlier? This would probably have caused alarm and would have 
allowed the decision-makers to deal with this in any number of 
ways, with any number of decisions. (Evaluation does not tell 
the decision-makers what decisions to make or what caused the 
conditions necessitating the decisions. Evaluation provides data 
to the decision-makers which they then use to make decisions or 
not, as the case may be.) 

They immediately took action to correct the situation, made 
several changes in their program, etc. In this instance, the 
technique would be used again very 8oon» perhaps 1 or 2 weeks later. 

In other words, this has all tried to say that how often a 
technique is used depends on the needs and decisions of the 
decisions-makers. A decision-maker should then be wary of the 
ev? \uator who wants to simply give a post-test. Suppose in the 
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above exainple, a post-test were ^iven in January or in June and 
it found that only 10 things were displayo.c^ If school vjere out 
for the summer in June, it would have been much too late to do 
anything and it might have indicated th-it this particular poal 
had been inadequately met, in fact it had not been met at all. 
If it had been done in January, half the year had gone by, vith a 
situation existing which really needed change. It is important, 
therefore, not to rely on such rules o^ thunb as post-tests. 
Seldo- if ever will such data collected be of a^eat decision makinp 
utility . 

(?!ote again that in the examole, r.iven, direct, natural and 
unobtrusive measurement was done. A questionnaire was not riven 
to the teachers to ask them what they did. Observation was 
carried out to determine it.) 

Implementation of measurement techniques should reflect 
decis ion-naker needs and decisions nade . 

It should also be remembered that thp frequency of use o"^ 
a technique will vary from technique to technique, as well as 
for the same technique. Therefore the decision-maker should not 
expect all the techniques to be administered or implemented on 
the oame time schedule or with the same frequency. This would 
not be efficient, or focused. Such a ripid pattern o^ collectinp: 
data would not vield the most effective information. (The most 
effective information is that which is there when you need it, in 
the amount you need it, and where you need it. Collecting; all 
the range of information all the time as would happen if all 
techniques were used the same would not meet this definition of 
effective. In fact, such an approach to measurement is costlv 
and a waste, both in time and energy and money.) 

Exactly when and how often a technique is to be used is a 
flexible situation. The deci sion -maker who wants the most effec- 
tive evaluation should expect a flexible schedule of collectinp 
data and should raise questions if the evaluator wants to admin- 
ister or implement techniques with the same frequency and in the 
sane time pattern. 

Sajnp^HjTj^: Another criterion about which the decision-maker 
should expect to interact with the evaluator is that of samolinp. 
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Sanplin;> becomes a very important criterion vhen one reaches the 
stage of collecting data ( implementinfr observational techniques). 
The evaluator should present any samplinr plan or procedures to 
the decision-maker in order to determine v^hether the nlan has 
decision-maker validity. The decis ion -maker should expect such 
an event to happen. 

'/hat is sar.pling? Samoling is nickint^ a nu^^.ber suLiects 
fror. a larger group of then. For exairple, if there are l,00n 
studer. ts in a school and one wished to detemine how ^any v/ere 
boys and how many v^ere girls (assuming we didn't hrtve this infor- 
mation) a sample mi^^.ht be taken all from the population (i.e., all 
1,000 of them). This sa-^ple might be 10^. (it is cheaner to only 
deal with 100 than 1000 in terns of time, r.oney , etc,) On the basis 
of randomly choosing a sample of 100, we find 55 ;rirls and 45 boys. 
V^e mi<3ht then, on the basis of this, estimate what tho percentage 
of each sex is in the whole population, 55% to 45^. 

This is a simplistic example to show tha: fror^. a smaller 
sample, it is possible to estimate some-hinr. about th'^ larger 
population. If a population of students, or subjects to be ob- 
served is large, then some kind of sampling should be done in 
order to reduce cost. Obrervinr all the subjects in a-Dopulatlon 
is often expensive. This expense mip;ht he wasteful because samo- 
ling (when done scientifically and carefully) can yeld the same 
in f ornat ion , or a good approximation of it. which a census of the 
whole population would yield* In the 1972 national elections, a 
Gallup poll of only 1500 people was sufficient enou::h and reDresen- 
tative enough to show what the whole votinp population would do. 
In the sample approximately 60 or 61^6 said they would vote for Mr. 
i:ixon. In reality, this percentage was almost exactly correct. 

Sampling is done to save time and money and effort. Sampling 
is also done when it is impossible to find out a piece of informa- 
tion from all the subjects in a population (as in the example of 
the election,) There are two criteria within samplin^^ which the 
r' e cis ion-maker should look for: size and representativeness. 

If one were measuring a poal on fighting in a school of 600 
one would probably want to look at more than 6 students. A 
sarnie size of 6 from a population of 500 will probably be quite 
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inadequate. The size of the sample should be larr^e enouph th^^t 
the de cision-naker is willing to generalize fron the sample to 
the population. 'fould a decision-maker generalize about 600 
students from a sample size of 6? It is unlikely. 

On the other hand, is it necessary to observe all 600 stu- 
dents to ^et an estimate of the anount of ficjhting ^oinp on in 
the school? Again, it is unlikely. A sanple o^' students or a 
sarr.ple of classrooms will probably yield data which is vali"^ 
eno'j-': to generalize to the school. 

The sainple size, therefore, should be lar^^e enough (or small 
enough) to maintain decision-maker validity vjithout overspending 
resources. If the decision-maker feels that the data which will 
be gathered from the samole will reflect the actual level of goal 
attainment in the population as a vrhole, then the sample size is 
sufficient . 

(There are certain scientific principles p-overninn samplinR 
and it nay be that just decis ion -maker validity may not be 

scientific' enough to justify certain e;enerali zat ions . The 
dec is i on-naker should expect the evaluator to point out such 
principles, in simple English durin^^ a discussion on sampling:). 
However, if hiving to apply too many principles jeopardizes 
decision-maker validity to the extent that the decision-maker 
feels data to be gathered will be useless to him, then decision- 
maker validity has been invalidated" and the decis ion -maker and 
evaluator need to discuss the problem. There is no sense in 
gathering data which no one will use in decision making.) 

The second criterion the decision-maVer shjuld consider is 
th-it of the representativeness of the sample. Goinr back to the 
example of fip^hting in the school, it may be that the size o^ the 
sample has decision-maker validity, but that the reoresentat i ve- 
ness of where that sample is to be taken does not. Let's say that 
the size has been determined to be 60 students. If the evaluator 
has designed a sampling plan whereby all these 60 students are 
^reshmen, when the school has four grades, then this plan is 
clearly not representative. If, however, the goal was held for 
only freshmen, then a sample of 60 freshmen would be very repre- 
sentative . 



If the sampling plan calls for selectinv"^ students from onlv 
social studies or only from industrial artf^ , when the goal is 
held for English also, then the {)lan is not representative. The 
decis ion -maker , then, should carefully iud<^e whether the sample 
is going to be representative. If he feels it is not, he should 
raise this point with the evaluator. 

In the final instance, it is the decis ion-naker who will 
use data for his decision making. It is the decision-maker who 
will have to generalize from data gathered from a sample to the 
whole polulation. To do this, he will have to carefully assess 
the size and representativeness of the sample. 
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vi ev;: Collect in p; Data 

(1) Is each technique dealt with ind i virtual ly with resoect 
to how often and uhen it vill collect data? 

(2) Does the schedule for collecting data nrovide for 
flexibility such that this schedule can be chan?5:ed 
(anywhere from more often to less often depending upon 
the nature of the data collecter^?) 

(3) Has the evaluator discussed the sanole and samplinp 
procedures with you to deter^iine your d ecis ion-maker 
validity? 

(U) Are you satisfied that the sancle tc be selected is 
representative of the larger poculaxion? 

(5) Are you satisfied that the sample to be selected is 
large enough to generalize to the lar^.er pooulation? 
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IX. HAVIIJG EVALUATION DATA REPOPTED TO THE DECISIOM-makER 

>/hen is the data reported?-' This very inportant question is 
one which is usually not addressed directly in evaluation and yet it 
is a crucial problem to consider. In nany evaluations which have 
been done, the data are collected at one point in tiiv.e and then 
the evaluator has cogitated, analyzed, summarized, synthesized, 
anl interpreted the data all at the same tir.e, foilo\;inr which he 
has written a report which is then delivered to the d e c i s i on - mak er 
quite often well after the need for evaluation data has tJ^-^ssed, 
e.^., in August, three months after the proiect has ended at least 
for the summer. Or in Septenber, two month- a^ter the in-service 
workshop has been conducted. 

This problem renortinp; data well afcer it is needed is 
one of the reasons evaluation has rotten a bad nai^^e and one reason 
that many people have criticized e\aluation .^z heinp less than 
useful. ^'hat has to be dene is that data need to be collected and 
reported as they ai e needed, not in one iumo sum at some terminal 
point in a project or enterprise. In the previous section which 
discussed data collection, the point was made that in some cases 
the same set of data may need to be collected several tines, es- 
pecially when changes have been made in order to more likely reach 
a p;oal. Uot to have the data reported until the en<^ of :hat class 
year will mean that further decisions to make chanpes i^ they are 
needed can not be made and the purpose of evaluation immediately 
becor-es less than being met. If data are not reported until the 
end of the year, for example, a decision to mak.e a chan?;e or not 
to make a chanp^e can not be madt on the basis o^ data. It is quite 
likely, in the example given of displayinr children's thinps . that 
even the need for making a decision would not come into the open. 

To be truly effective, then, data for decision -ma kin p. need to 
be reported as closely as possible to when they have been collected. 
A'^so, the evaluator should be ready to collect the same data again 
in a short period of tir-e if neces.sary. ?^tp. collection h^s to be 
responsive to d ec i s i on - make r needs. 
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^^ha t is to be rer ' ^''^t ec' '? A^ain, thi? "-rrht se.-^-^ to be a ques- 
tion with a very obvious answer but when it is considered care^ 
fully, it will be seen that it is really nuch nore co-nplex than is 
usually thought. 

''The data are reported''. This is the answer. But, v;hat 
comprises the data? Data can be considered as the information 
gathered by the observational technique and they v/ill probably have 
some numbers or figures or charts. This is what nany evaluations 
report as data. It is really a narrov/ de'^inition b^c^xi'^e there 
are many other things which should be reported in conjunction with 
these number ''data' which become important in the decision makinp: 
process . 

A data report should include nauy thinr.s besides the numbers. 
It should contain the followin,?- things: 

1. The name of the decision-maker for whon these r^rticular 
data were collected. It ha^ heen pointed out many tirr^es 
that there are many decis icn-makors in an enterprise. If 
the primary decision-maker for whom these data were col- 
lected is the chairperson the m^th d e:^ artment , then 
this information should appear on the data reoort . 
''Isn^t this obvious?'' one mi^ht ask? If is is, fine; if 
it is not, then it should be. The other decis ion^makers 
in the enterprise, e.p,, the math teachers or the 
assistant superintendent for curriculum and instruction 
or the principals v;ill probably, at one point or another 
also be given a copy of the data and it is essential 
that these other decis ion-*makers know for v/hom and from 
whose perspective the data were collected. (Different 
decision-makers need different kinds of data. Reporting 
the data of the chairperson, to the princioal if he does 
not know whose data it is, is likely to not view the 
data as meeting his needs. The Doint is ^ they may very 
I'.kely not :neet his needs because they v;ere collected 

for someone else. This is why such labelin<^, is important). 

2. The name of the goal and its imnortance (or prioritv) 
to the particular decision-maker. Take, for exanole, 
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the earlier discussion the goal havinR children's 
things displayed". This intent was one of the onerational 
components of the more general goal to have an affective 
climate in the pro^^ram ' . (The '^display ' intent vras only 
one of the many, nany items • The data report ^or this 
particular item then shou.ld inclu^-^e the fact that this 
was part of the larger goal and tyiat this l?.rr;er roal 
was the ^'1 goal this particular de ci s i on -maker (the staff 
of U) held for the program. 



The importance of the operational component. The reader 
migl c be thinking at this point. 'But havinp children's 
things displayed- , does not seem to me to be a very im- 
portant part of ''affective climatfS . The data report 
should also contain then the importance o^ the operational 
component to the decision-maker for './horn it is being 
collected. For example, in this case, the report night 
contain "this component of display was ranked as number 
27 of the 70 components of the goal ^affective climate"'. 
This information then Rives other decision-makers infor- 
mation for their decision making needs. 



U. The name (and description if appropriate) of the obser- 
vational technique used to collect the data. 

5. The date of the data collection (or dates if approoriete) 
and the place, e.g,, September 17, 22 and 28 in Mr. 
Teacher's class and Miss Teacher's class. 



6. The actual data, presented in terms which the ^^ecision- 
maker for v;hom it is being collected can use and under- 
stand. 



These 6 items are important items vrhich should be cart of a 
report on data. They are items which the decision - m.aker should 
expect. Guch information clarifies the report and makes the data 
(in many cases) more effective, both to the r^rimary decision- 
maker and other decis ion^makers of the enterprise. 
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Revi ev/: Data Report in/; 

(1) Is the data reported when it is ne< .:pr^? In the anovnt 
needed? On the appropriate Ito,'^'^ ne^^^e-:? 

(2) Does the report include more than iust a few nvirhers 
end statistics? 

(3) Specifically, does the report include: 

a. the name of the person(s) for vrhon this particular 
set of data were collected? 

b. the name of the poal and the imooT^tance of the 
goal which this data is beinr ocllected to -'•easurn? 

c. tho iTiportance of this particular operational 
coTiponent to the larger f.oal? 

d. the nane and description of the observational 
technique? 

e. the date, time and place of riata collection? 
the data? 

(^) Are the dfita presented in an understandable fashion? 
Such that they can be used and understood by the 
decision-maker for whom they were collected? 

These ar^e criteria a decision-maker should look for and ex- 
oect in a reoort of evaluation data. 
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• hat a report of c^ata should not hove 



Just as theie are thinps whicri a ue cis i on - r.cjk ^r should ex- 
pect P.T\fl look for in a report on data, there a^.^e also things he 
^"^'py.--^ L*?-^ find in such a report and if he doos find such thin,^s, 
lie si:ould be skeptical about them and qiiestion the evaluator about 
including things which shouldn^t be included. 

Tr.e decision-maker should not find, within such a report, 
decisions made by the evaluator on the data. ""decisions about the 
data, int eroret ation about the data, how s i q:r i -Fi cant are the data: 
these are properly made by the decis ion-rrakr.- . The evalu-'^tor 
should not v/rite such things as 'These are rood, the nroject 
should continue doing . . . Or ^ 'These are bad, the oroject 
should chanr.e what it is doinp, and do this ... Such conclu- 

sions and recommendations are outside the proo'^r realm o-^ the 
evaluator. Such inferences are for the dec i s i on-^-^r-^ker to draw. 

The report should not contain evaluator biases in the form 
of passing his personal judgments about the data or the techniques 
or the observations. Such personal likes and dislihiso"^ an eval- 
uator are outside the scope of evaluation. (If a decision-maker 
v/ishes to hire someone who will come in and ruaVe svich statements, 
then he should Uo so. However, such activity should not be called 
evaluation but judpiment. 

The report should not_ contain information from the evaluator 
which tries to influence the program in one direction or another; 
Khich tries to have specific or particular decisions made* about 
the program^s adequacies or inadequacies. These are in the domain 
of the decision-maker's responsibility. Again, if a decision- 
maker wants to hire someone to come in and make decisions, or 
reco!^mend decisions then he should hire someone to do so^ but he 
should not call if evaluation. 

The report should also not contain a section entitled 
"Commendations for the same reasons cited above. Many evaluation 
reports contain a list of things which are commended" for the only 
apparent reason that the evaluator liked them. Such activities 
ar*^ outside the legitimate scope of evaluation. 



The sarTie can bo said of a Sr ction in r^anv evaluc^tion reports 
entitled r e conmen da t i ons " . Such sections should be deleted for 
these are the responsibilities of the d ec i s i on- mak er , Everyone 
likes to be commended but T,any (if not most) do ci s i on- nak ers vould 
arf^ue with such recommendat ions ' v;hich of necessity "^ust reflect 
a shortcor.ing at least as seen by soneone. A k i nderf^art en teacher 
will not argue with those things she is comnended ^cr , but in at 
least one evaluation v/here the evaluator overs teppe-^. his bounds and 
included a section of Re con-.end a t i ons , the t ch e r , vho was the 
prip.ary decision-maker for this particular evaluation, disnuted 
each and every recommendation v/ith such rosponses as, "He d-esn^t 
understand kinderparten children • ''Me isn't an exrert in early 
childhood', 'He doesn't understand ooen c 1 3 s r oc-. . ' e recom- 
mends such and such which is not at all a ,eoal of the program . 

::hen an evaluator moves into the realri of ' recommendation;-'- 
and commendations'', he moves out o^ i;roper realm of evaluation 

and into the realm of decision-iiaker for --^n sjr.terprise of which 
he is not in fact a legitimate decision-maker, A d e c i s i ^n - mak er 
should bev;are the evaluator who want to, or doos , ^et into this 
area of decision making, for it is precisely that., decision makinp,. 
Decision making is not evaluation. ^valuation r,hould serve deci- 
sion making and it can do this far better by not tryin,<3 to cooDt 
decision makinr. i^^t by provi-iin?:' data to proper and ler^itimate 
decision -makers . 
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Reviov7j \'hat a data rencrt should not h'*vc 



(1) Does the report have decisions (personal) of the evalr.a tor ? 
( It shouldn ' t . ) 



(2) Does the report have the person-il likes and dislikes 
of the evaJuator? (It shouldn't.) 

(3) Does the report contain reco'^iincndations of the evaluator 
about the pro,';ram^ its direction, content, and so on? 

( It shouldn ^ t . ) 



( Does the report have a ' Comnendat i ons section and a 

■Recommendations" section v^ritten by the evaluator? 
( It shouldn ' t . ) 
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r::designimg the evaluati; 



Redesigning the evaluation is an option which occurs only 
in certain circumstances. Ordinarily, the dec ision-naker would 
not expect redesi^^n to be part of every evaluation but the topic 
will be discussed here so that the de c is i on -rak er ni.eht know what 
a redcsirn should include and when it mip^ht be done. 

If the evaluation has been done properly to this point, with 
the interaction of d e c i s i on - make r and evaluRto-- and i^ the evalua- 
tor has bp en carefully fulfillinr his role and n^ot confusing his 
role v^ith that of a de c i s i on - na k and if thp d e c i s i on- mri kc r is 
fulfilling his role conscientiously, then there \j \ 11 proh::bly be 
no need for a redesign 3 ect i on per se . Each step of the Process, 
if the reader will remenber, has a kind of redesign part to it. 
A step is not complete unless it has beeii s a t i s ^^ac t or i ly agreed 
to by the d oc i s i on - na ke r and evaluator. For cxar^nlo, durinp^ the 
goals process, the de c i s i on -ma k er must decide on which poals to 
include and which to omit. He must also decide on a priority 
order (v/ith tho evaluator providing- the evaluation exT:^ertise 
necossary to help the de c i s i on - ma k e r ) . If those processes are 
r.one through and the decision- mak er says, No, that is net the 
goals list I really hold, or "No, that is not the priority order 
of my goals,' then that particular section is recycled on the spot. 
This could be called a redesign of the roal'^ nrocess. 

The same thine; is provided for in each process of the eval- 
uation. At least, it should b.?. A section i>"^ recycled or re- 
designed as a section until it is satisfactory. (Again, this is 
not likely to be necessriry if the d ^ r i s io n - ma k e r has been actively 
and conscientiously involved in the evaluation design as he should 
have been ) . 

'Miat are some circumstances under which relesign of the en- 
tire evaluation might be needed? 

Redesi^^n might occur if or v;hen: 

1, The program or project changes dramatically or drastically 
For exariole, the decision-ir^aker v- it bin the project may 
leave, re sign, die or be promoted, in effect changing 
the person{s) with whom the evaluator has been working 
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and for whon the '^'valuation has been desi^^ned. This v/ould 
necessitate redesigning the evalua t i '^i^ , 



7he emphasis of the program changes (i.e., tiie p:oals 
change). During the course of a project or enterprise, 
^.oals are very likely to chan^i^. If this occurs, then 
redesign is necessary in order to reflect a chanpe in 
goals or in priority of goals. This will in turn neces- 
sitate different observational techniques beinr. desi.^ned, 
different data being collected, etc. 



The enterprise experiences a break" or Rap' betv/een 
one part of its opei^gtion and another. Tnis night occur 
in a Title III project, for example, '/hich has been 
funded for three years. At the end o' the first year, a 
decision r^ight be made, or decisions made, which in turn 
would necessitate chanp;inr the evaluation. These deci- 
sions could deal with personnel chanr;e, rro^ram chancres, 
financial changes, content changes, etc. 



The enterprise is a long-term one. An example of tills 
might be any part of a school system, e.n-* math cur- 
riculum, English department and so on. In this instance, 
it is a sound idea to have an evaluation redesif^n stage 
built in. So many variables can change durinp the course 
of an enterprise, especially a lonp,-tern one that it 
really is necessary to provide for redes i p;n in evaluation. 

A confl'^ct, misunders tandin^T or some similar pro])lem, 
occurs between the evaluator and decision maker. This 
might happen for examnle if the two parties diri not under- 
stand their purposes and functions r^\irinp the first step 
of initiating evaluation and that misunderstanding did 
not become apparent until some time during the evaluation. 
Such misunderstandings could include or focus on: the 
purpose of evaluation, with one party v/anting someone to 
make decisions and the evaluator desip.ninp an evaluation 
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to provide data to the enterprise de cisicn-makars . An- 
other example might be that in the initial phase of the 
evaluation, the v/rong or incorrect decision-maker was 
identified. The decision-maker who actually makes the 
decisions was somehow not properly identified. This in 
turn would mean that the evaluation has been»desi}3:ned to 
pr ovid e dat a to the wron p person and th us a redesic;n 
vjould be necessitated. 

Interpersonal relations-personality problems: As with any 
endeavor, these kinds of problems can ent e r the pi cture 
and could cause changes to be made. For exam.Tile, the 
evaluator might have a value conflict with the decision- 
maker causing the evaluator to desire to leave the nro- 
ject. On the other hand, the decision-maker may experience 
value conflicts or personality problems with the eval- 
uator and might cause him to ask the evaluator to l^ave. 
(A reminder might be made here that in preparing the 
contract, there should be stipulations allowing for this 
to occur without penalties to either party. A termina- 
tion clause should be writeen in for the mutual benefit 
of both parties should the example just given arise. The 
decision-maker does not want to be saddled with a person 
whor' it turns out is completely incompatible with the 
needs of the d ec i s ion -make r . Conversely an evaluator 
can not provide the most efficient evaluation design i^ he 
feels that there are incompatible dif "Ference?> between 
hirriself and the decision-maker.) 



Review: Redesigninr the e v aluation 

(1) Redesir^n may or may not be part of every evaluation. 

(2) If redesign is necessary, it riay he so for anv number 
of reasons. It would be impossible to detail them all 
here. They are the same kinds of reasons which can 
cause problems in any educational enterprise. 

(3) If redesign is necessary, then it should :f^ollow the 
Game guidelines provided herein for a good evaluation. 

(^) Finally, redesign is goinr; to corjt additional resources 
especially time. The dacision-maker should consider 
this before makinf^ the decision to have a redesign 
carried out. 

(5) In the final say, it is the decision-maker who decides 
to have the evaluation redesigned or not. 

Observation of the evaluation process by the decision-maker 
using these guidelines (provided throughout this booklet) may 
provide the basis on which to make the decision that a redesign 
is necessary. This could happen as soon as difficulty occurs in 
the evaluation process, rather than finding out during the last 
month of the evaluation that a redesign is needed. However, such 
a decision to redesign when difficulty arises can only happen if 
the de cis ion -maker has been checking the process all along the 
way. It is suggested that the guidelines provided herein could 
serve as criteria to check the evaluation process throughout, 
not when it is done. 
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x:. EVAL'JATio:: or ev/^l = ' ••ti '^^ 

EvaluatinF the evaluation is part of evaluation. Vet very 
fe'.; evaluations which have been done have had provisions for 
evaluarinp thenselves. Ir. fact, most evaluations which have been 
done in trie past usually terminate vfith a Final Reoort, when it is 
too late to systenatically evaluate that Final Peoort. 

Ore very important thinp v/hich a d e c i s i on - maker should expect 
is to :.ave sone provisions made for an evalur^tion o^ the evaluation, 
As v;ith all the other processes of evaluation which this booklet 
has discussed, the decision-maker must actively rarticinate in 
this process . 

If an evaluation is accomplishing its -Mrpose, that is, pro- 
viding valid data to the d e c is ion mak er for his d--^cision making 
needs, then certain events are occurring and certain events are 
not oc curr in £ * 

1. Data provided tc the d e c i s i on - r.ak e r are actually used 
by hip (her, therrO in nakinp; decisions. 

2. The evaluation is efficient: All the data collected for 
a particular decision-maker are used by him. . To -^he 
extent that data are collected and provided and not used, 
the evaluation has not met its puroose . 

3. The evaluation is complete: Of the decisions made by a 
decision-rraker relative to a p^irticular program or 
enterprise, as many as oossible are made with data oro- 
viaed by the evaluation. 

^. The evaluation is focused: If data can not be provided 
(because of lack of sufficient resources like time and 
money) for all the decisions, then it should be provided 
for the most important decisions. 
These three criteria - efficiency, comipleteness and focus - 
can be applied by the dec is ion -maker to the evaluation in order 
for him to determine the extent to which the evaluation is meeting 
its purpose of providing data for decision making. 

It is probably impossible that any evaluation will cc-mpletely 
me^t these criteria. There are many reasons for this. Firct, 
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evaluation efforts nay be be^^un too lete In the course of the 
pro^^rrim or project in order for data collected to neet the 
criteria. An evaluation can not fully meet the criteria i^ it is 
not begun until half-way through the project. 

Second, resources vfill probably never be sufficient to allow 
the evaluation to completely meet the criteria. It is nrobablv 
in-)ossible to collect all the data, needed by all tho decision- 
niakcrs of a project to neet a_ll their decision T7,akinr needs, 
because the cost of doing this would be prohibitive. This implies 
certain things then v/hich the decision-maker shoul^-^ take into con- 
sideration in evaluating the evaluatio.i. The Hecis ion-maker must 
be cognizant of the amount of resources committed to the evaluation 
because resources deternin^:* the scooe o^ the evaluation. He must 
remember that not all the data can be provi-'ed to all the decision- 
maker3 for v:hom it mi^.^ht be desirable. That is whv , durinr the 
course of the evaluation, the Pjjj^_ary_ jecj^s ion -nakers are identi- 
fied and prioritized so that those persons most needinr information 
^T^i^bt get it. That is why the most important g;oals of the primary 
decision-makers are identified so that they -'.ight get indorsation 
on their most important needs or p.oals . If durinr, the course o^ 
the evaluation even one of these was done incorrect ly / the eval- 
uation will become less efficient, less conolete and less ^ocusad. 

One i:ay a d -3c i s ion - ma ker mipht collect information -^or him- 
self so that he might evaluate the evaluation in terms his oun 
needs is to keep a log of decisions made relative to the program 
evaluated. Ideally evaluation and planning o^ the program occur 
at the same time, prior to the beginning o*' the program. If they 
are not or can not be, the d ec i s ion - ma ke r should remember that this 
will a^-Pect the evaluation of evaluation. For those decisions, he 
should note their relative importance to him. Then, he should 
assess whether and how much data was provided to him for those 
important decisions, and was it provided when he needed it. In 
other words, apply the three criteria. 

v?hat are some other things a decision-maker might consider 
in performing an evaluation of the evaluation? Evaluation .should 
n ot interfere v;ith the enterprise's a c com nl i sh in g its goals (un- 
les'^ the goals are in conflict with one another and then this 
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beco-,os not a problem or fault the c-valuator but a decision 
mnkin, problen.) m fact, evaluation should help an enterprise 
to accomplish its goals by having inforr.ation sy s t e na t i c al ly pro- 
vider durin^ the course of that enterprise, su" that the r^ecision- 
Tr.akers o- that enterprise can use it in their decision making. 
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Review: Evaluation of Evalu a t i on 

(1) Is the evaluation orovidinf^ data ^or your decision 
nakin^ needs relative to the identi*^ied en"^ernrise? 

(2) Given the scope and resources of the e va lu -"^ t i on : 

- Is the evaluation efficient? 

- Is the evaluation conplete? 

- Is the evaluation focused? 

(3) Are you keeping a Icf of decisions you nake relative 

to the identified resources in order to be able to assess 
points TPentioned above? 

(u) Does the evaluation or evaluator interfere with you 

and your enterprise achievin;;^ its r^^^als? (They shouldn^t) 

(5) Finally, a person usinr this guide can evaluate the 
evaluation in terns of its Darts, e.r., the contract 
phase, goals process^ parts nrocess . and so on, i^ he 
monitors the evaluation usinp; the criteria provided ^or 
each section. This would be done in addition to keeoinp 
a lop; of decisions (in 3 above). 
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XII. WHEN RESOURCES FOR THE EVALUATION ART PEALLY SMALL, 

WHAT DO YOU DO? 

This booklet has tried to present a conorehens i ve picture of 
the complex task of evaluation. However, the reader may have 
gotten the impression that "Uell, this is all fine and p;ood, but 
I have very few resources and I just can't buy all of this. 

Resources will always limit the scope of the evaluation. 
Lir.ited resources will have to limit the scope but do not have to 
exclude doing evaluation entirely. Limited resources simply will 
mean that the evaluation will have to be more efficient and more 
focused than unlimited resources. 

The evaluation must in fact fit, from bepinninp, to end, 
starting to deliver usable data within the resources that are 
actually available to do the job. Therefore resource allocation 
becomes a very important part of th^, ^valuation. All the resources 
can't be spent on any one part of the evaluation, e . . , identifying 
goals, or doina a parts analysis. If resources are small, really 
small, then whet is needed is as complete goals process as 
possible within limits , as complete a parts process as possible 
within limits , and so on. 

Limited resources will mean probably dealing with only one 
(the most important or primary) decision-maker of an enterprise 
It Mill mean noc doing a lot of tests of completeness in the 7oals 
process. Possibly, because of the focused nature of the evaluation 
(on a very specific and well defined enterprise) the parts process 
will be eliminated entirely. 

Limited resources will also mean not opera t i ona 1 i zi n ^ all 
the coals as completely as possible. It will probably mean 
operat ionalizinp just the most important goal of the most important 
decision-maker. Throughout the evaluation, there will be short 
cuts and shortened forms of the processes. However, the basic 

evaluation, even if in shortened 

^^^^^ not mean that a decision- 
focused and useful evaluation. 
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processes should still be in the 
form . 

Even very limited resources 
maker has to forego a systematic. 
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An evaluation is alv/ays shi^oed bv tha rttsources. "ver: when abun- 
dant or limitless resources are available there is a need for a 
focus ing of it . 

Hv havinrr some gui^felines to use, a deci ion-maker can be 
aware of the shortcuts and shortcomings of an evaluation as well 
as the strong points and advantages an evaluation. Because 
there ere limited resources does not mean that the decision-maker 
should reject evaluation. In the final instancf?, evaluation, or 
prcviding data for de c i s i on - n^ake r s , is meant to helo the decision 
maker, not hinder him. The suf^pestions rrovi^^en herein are 
intended to aid the de c i s i on - make r i^^ the evaliiative process. 



XIII. A GLOSSARY QT TERMS 
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B>j h a v i o x -^a 1 Qbj[^e_ctjj{^ : a statement of what you want someone 
( usually a learner) to acconolish, stated in very spe- 
cific ,behavioralterms* 

£5jLi^. f.^il ^^ A.^ is ion iTiaking : Thii; is the st ite:?.ent of tne pur- 
pose of educational evaluation, first set forth by Cron- 
bach in 1963 and now widely held by the leading experts 
in the field, including S t uf f lebe am , Hutchinson, Guba, 
V^orthen, Provus and so on. 

Decision Maker : Any person who in sor.o v;ay makes a decision 
about a particular project, progran^, endeavor or enter- 
prise. For a school, exariples would be; students, par- 
ents, teachers, administrators, staff, school committee, 
^ t c . 



Ent erpr i se : That about which data is to be collected, that 
which is to be evaluated: can range from a single lec- 
ture to a whole program or project (e»g., Title I or 
III), to a school, to a national program. 

Ev^lua_t_ion^: the act of identifying, collecrir^-, and reporting 
data to decision makers for their decision making needs. 

tHJ-Jil. Concept^: Anything which is not directly observable or 
measurable is a fuzzy concept; a goal which is nebu- 
lous, vague, general, e*g,, good citizen, autonomous 
learner, self-actualization, 

PJ1?Jl' statement of intent or an aspiration, something 

you want to accomplish: usually stated in fuzzy terms. 

Me thodo l ogy : A standardized, opera tional ized , systematic 
set of rules and procedures for accomplishing a de- 
fined purpose . 

'''' generalized, non-specific set of general rules- 
of-thumb or guidelines for accomplishing a purpose, 
a set of non-operational, fuzzy procedures for doing 
something . 

Observa tional Technique : Something with which to collect 
data , not just limited to a "''test^'. 

Ope ra tionalize : To take a fuzzy concept and sysiemat icaHy 
put it into its specific, concrete, observable, mea- 
surable states . 
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GLOSSARY OF TERMS 
( conr * d ) 



Prioritize : To put in some kir.d of order, e.g., putting a 
list of items in order of most important to least im- 
portant or from first occurring in time to last occurr- 
ing in time. 

Resources : A term referring to money, time, staff, m.ater- 
ials, space, expertise: those things which are needed 
ro carry out an evaluation. 
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